Snapshot Hyperspectral Camera Research Articles

Dual-camera laparoscopic imaging with super-resolution reconstruction for intraoperative hyperspectral image guidance.

Laparoscopic and robotic surgery, as one type of minimally invasive surgery (MIS), has gained popularity due to the improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery. Hyperspectral imaging (HSI) is an emerging medical imaging modality, which has proved useful for intraoperative image guidance. Snapshot hyperspectral cameras are ideal for intraoperative laparoscopic imaging because of their compact size and light weight, but low spatial resolution can be a limitation. In this work, we developed a dual-camera laparoscopic imaging system that consists of a high-resolution color camera and a snapshot hyperspectral camera, and we employed super-resolution reconstruction to fuse the images from both cameras to generate high-resolution hyperspectral images. The experimental results show that our method can significantly improve the resolution of hyperspectral images without compromising the image quality or spectral signatures. The proposed super-resolution reconstruction method is promising to promote the employment of high-speed hyperspectral imaging in laparoscopic surgery.

Compact snapshot hyperspectral camera for ophthalmology

Hyperspectral imaging is an emerging technique that allows measurement of spectral absorption at each point of a scene, thus offering the capability to identify and characterize important biomarkers for clinical practice and therapeutic research, as well as enhancing image identification of important structures. So far, few hyperspectral cameras have been used for retinal scanning because of the need to acquire the image in a fraction of a second. Here, and to the best of our knowledge, we present a novel concept of a snapshot hyperspectral camera suited for retinal imaging. We demonstrate the technique by presenting the optical density spectrum of a healthy patient’s retina in the 450–700 nm range, together with the spectral response of several retinal features.

A High-Resolution Hyperspectral Imaging System for the Retina.

In this study, we developed an imaging system that can acquire and produce high-resolution hyperspectral images of the retina. Our system combines the view from a high-resolution RGB camera and a snapshot hyperspectral camera together. The method is fast and can be constructed into a compact imaging device. We tested our system by imaging a calibrated color chart, biological tissues ex vivo, and a phantom of the human retina. By using image pansharpening methods, we were able to produce a high-resolution hyperspectral image. The images from the hyperspectral camera alone have a spatial resolution of 0.2 mm/pixel, whereas the pansharpened images have a spatial resolution of 0.1 mm/pixel, a 2x increase in spatial resolution. Our method has the potential to capture images of the retina rapidly. Our method preserves both the spatial and spectral fidelity, as shown by comparing the original hyperspectral images with the pansharpened images. The high-resolution hyperspectral imaging device can have a variety of applications in retina examinations.

An open-source machine-learning application for predicting pixel-to-pixel NDVI regression from RGB calibrated images

The Normalized Difference Vegetation Index (NDVI) is the most common index to measure vegetation in agriculture and create classified prescription maps used for several purposes. It is obtained as the ratio between information from the visible and near infrared spectral bands. The calculation of NDVI often uses very expensive hyperspectral and multispectral optical sensors. This study aims to develop an efficient model to extract NDVI data from RGB images. During the work were acquired images, of different plant species at different vegetation status, through a snapshot hyperspectral camera (Specim IQ) featuring natively superimposed RGB sensor whose images were calibrated a posteriori (sRGB). NDVI was predicted from sRGB images using a Shallow-regressive neural network that performs a pixel-pixel regression. The model has been then tested on around 1000 drone images acquired by a 6x Sentera sensor to test the efficiency of the applied model. The model has been used to estimate the numerical value of NDVI and the classes produced by a clustering method (k-means). The results of this work showed that the calibrated images had a very high correlation with the NDVI in validation (r = 0.91), maintaining good performances (r = 0.71) when applied to a set of data acquired with a different non-co-registered sensor. Therefore, for the first time this work has shown that the health of vegetation through the NDVI could be calculated using unexpensive RGB device adopting a pixel-pixel regression AI approach. The approach shows its importance especially for small-sized farms where profit would not allow for the budget to access multispectral cameras and heavy carriers (i.e., UAV).

Aperture-encoded snapshot hyperspectral imaging with a lensless camera

We present a lensless snapshot hyperspectral camera that is capable of hyperspectral imaging over a broad spectrum using a compact and low-cost hardware configuration. We leverage the multiplexing capability of a lensless camera, a novel type of computational imaging device that replaces the lens with a thin mask. Our device utilizes a linear variable filter and a phase mask to encode spectral information onto a monochromatic image sensor, enabling recovery of hyperspectral image stacks from a single measurement by utilizing spectral information encoded in different parts of the 2D point spread function. We perform spectral calibration using a reference color chart and verify the prototype device’s spectral and spatial resolution, as well as its imaging field of view. We report on the design and construction of the device, the image reconstruction algorithm, and spectral calibration methods and present hyperspectral images ranging from 410 to 800 nm obtained with our prototype device.

Automated Polarized Hyperspectral Imaging (PHSI) for ex-vivo and in-vivo Tissue Assessment.

Polarized light interactions with biological tissues can reveal information regarding tissue structure, while spectral characteristics are closely related to tissue composition. An integration of both modalities in a compact system could better assist tissue assessment. This study aims to develop a polarized hyperspectral imaging (PHSI) system that fulfills both linearly and circularly polarized hyperspectral imaging for in vivo and ex vivo applications. The system is comprised of a white LED, two linear polarizers, two liquid crystal variable retarders (LCVRs), and a hyperspectral snapshot camera. The system was calibrated to compute the full Stokes polarimetry. For tissue differentiation, fresh ex vivo mouse tissue specimens from kidney, liver, spleen, muscle, lung, and salivary gland of mice were imaged. The spectra of three features, named degree of polarization (DOP), degree of linear polarization (DOLP), and degree of circular polarization (DOCP), were generated. A k-nearest neighbor (k-NN) classifier was trained with multi-class spectra and 5-fold cross validation. It was found that DOP better differentiates tissue with an average accuracy of 0.87. Additionally, support vector machine (SVM) classifiers were trained to differentiate between each two of the organs, and it was determined that DOLP better identified kidney, liver, and spleen, whereas DOCP better identified muscle and lung tissues. Then, the setup was employed to image in vivo human fingers with and without a blood occlusion to qualitatively estimate oxygen saturation. Preliminary results demonstrate that both DOLP and DOCP reveal a distinction of oxygen saturation states. These results demonstrate the feasibility of the PHSI system for distinguishing between optical properties of tissues, which has the potential to reveal disease-related information for diverse medical applications.

Design and realization of a miniaturized high resolution computed tomography imaging spectrometer

The computed tomography imaging spectrometer (CTIS) is a relatively unknown snapshot hyperspectral camera. It utilizes computational imaging approaches to gain the hyperspectral image from a spatio-spectral smeared sensor image. We present a strongly miniaturized system with a dimension of only 36 × 40.5 × 52.8 mm and a diagonal field of view of 29°. We achieve this using a Galilean beam expander and a combination of off-the-shelf lenses, a highly aspherical imaging system from a commercial smartphone, and a 13 MP monochrome smartphone image sensor. The reconstructed hyperspectral image has a spatial resolution of 400 × 300 pixel with 39 spectral channels.

Design and realization of a miniaturized high resolution computed tomography imaging spectrometer

The computed tomography imaging spectrometer (CTIS) is a relatively unknown snapshot hyperspectral camera. It utilizes computational imaging approaches to gain the hyperspectral image from a spatiospectral smeared sensor image. We present a strongly miniaturized system with a dimension of only 36mm x 40.5mm x 52.8mm and a diagonal field of view of 29°. We achieve this using a Galilean beam expander and a combination of off-the-shelf lenses, a highly aspherical imaging system from a commercial smartphone and a 13 MP monochrome smartphone image sensor. The reconstructed hyperspectral image has a spatial resolution of 400 x 300 pixel with 39 spectral channels.

Detection of cutaneous oxygen saturation using a novel snapshot hyperspectral camera: a feasibility study.

Tissue necrosis, a consequence of inadequate tissue oxygenation, is a common post-operative complication. As current surgical assessments are often limited to visual and tactile feedback, additional techniques that can aid in the interrogation of tissue viability are needed to improve patient outcomes. In this bi-institutional pilot study, the performance of a novel snapshot hyperspectral imaging camera to detect superficial cutaneous oxygen saturation (StO2) was evaluated. Healthy human volunteers were recruited at two participating centers. Cutaneous StO2 of the forearm was determined by a snapshot hyperspectral camera on two separate study days during occlusion-reperfusion of the brachial artery and after induction of local vasodilation. To calculate the blood StO2 at each pixel in the multispectral image, spectra were selected, and fitting was performed over wavelengths ranging from 470 to 950 nm. Quantitative detection of physiological changes in cutaneous StO2 levels was feasible in all sixteen volunteers. A significant (P<0.001) decrease in cutaneous StO2 levels from 78.3% (SD: 15.3) at baseline to 60.6% (SD: 19.8) at the end of occlusion phase was observed, although StO2 levels returned to baseline after five minutes. Mean cutaneous StO2 values were similar in the same subjects on separate study days (Pearson R2: 0.92 and 0.77, respectively) at both centers. Local vasodilation did not yield significant changes in cutaneous StO2 values. This pilot study demonstrated the feasibility of a snapshot hyperspectral camera for detecting quantitative physiological changes in cutaneous StO2 in normal human volunteers, and serves as a precursor for further validation in perioperative studies.

3D-printed miniature spectrometer for the visible range with a 100 × 100 μm&lt;sup&gt;2&lt;/sup&gt; footprint

The miniaturisation of spectroscopic measurement devices opens novel information channels for size critical applications such as endoscopy or consumer electronics. Computational spectrometers in the micrometre size range have been demonstrated, however, these are calibration sensitive and based on complex reconstruction algorithms. Herein we present an angle-insensitive 3D-printed miniature spectrometer with a direct separated spatial-spectral response. The spectrometer was fabricated via two-photon direct laser writing combined with a super-fine inkjet process. It has a volume of less than 100 × 100 × 300 μm3. Its tailored and chirped high-frequency grating enables strongly dispersive behaviour. The miniature spectrometer features a wavelength range of 200 nm in the visible range from 490 nm to 690 nm. It has a spectral resolution of 9.2 ± 1.1 nm at 532 nm and 17.8 ± 1.7 nm at a wavelength of 633 nm. Printing this spectrometer directly onto camera sensors is feasible and can be replicated for use as a macro-pixel of a snapshot hyperspectral camera.

Combination of snapshot hyperspectral retinal imaging and optical coherence tomography to identify Alzheimer\u2019s disease patients

IntroductionThe eye offers potential for the diagnosis of Alzheimer’s disease (AD) with retinal imaging techniques being explored to quantify amyloid accumulation and aspects of neurodegeneration. To assess these changes, this proof-of-concept study combined hyperspectral imaging and optical coherence tomography to build a classification model to differentiate between AD patients and controls.MethodsIn a memory clinic setting, patients with a diagnosis of clinically probable AD (n = 10) or biomarker-proven AD (n = 7) and controls (n = 22) underwent non-invasive retinal imaging with an easy-to-use hyperspectral snapshot camera that collects information from 16 spectral bands (460–620 nm, 10-nm bandwidth) in one capture. The individuals were also imaged using optical coherence tomography for assessing retinal nerve fiber layer thickness (RNFL). Dedicated image preprocessing analysis was followed by machine learning to discriminate between both groups.ResultsHyperspectral data and retinal nerve fiber layer thickness data were used in a linear discriminant classification model to discriminate between AD patients and controls. Nested leave-one-out cross-validation resulted in a fair accuracy, providing an area under the receiver operating characteristic curve of 0.74 (95% confidence interval [0.60–0.89]). Inner loop results showed that the inclusion of the RNFL features resulted in an improvement of the area under the receiver operating characteristic curve: for the most informative region assessed, the average area under the receiver operating characteristic curve was 0.70 (95% confidence interval [0.55, 0.86]) and 0.79 (95% confidence interval [0.65, 0.93]), respectively. The robust statistics used in this study reduces the risk of overfitting and partly compensates for the limited sample size.ConclusionsThis study in a memory-clinic-based cohort supports the potential of hyperspectral imaging and suggests an added value of combining retinal imaging modalities. Standardization and longitudinal data on fully amyloid-phenotyped cohorts are required to elucidate the relationship between retinal structure and cognitive function and to evaluate the robustness of the classification model.

Limitations of snapshot hyperspectral cameras to monitor plant response dynamics in stress-free conditions

Plants’ dynamic eco-physiological responses are vital to their productivity in continuously fluctuating conditions, such as those in agricultural fields. However, it is currently still very difficult to capture these responses at the field scale for phenotyping purposes. Advanced hyperspectral imaging tools are increasingly used in phenotyping, and have been applied to detect changes in plants in response to a specific treatment, phenological state or monitor its growth and development. Phenotyping has to evolve towards capturing dynamic behaviour under more subtle fluctuations in environmental conditions, without the presence of clear treatments or stresses. Therefore, we investigated the potential of hyperspectral imaging to capture dynamic behaviour of plants in stress-free conditions at a temporal resolution of seconds. Two growth chamber experiments were set up, in which strawberry plants and four different background materials, serving as controls, were monitored by a snapshot hyperspectral camera in variable conditions of light, temperature and relative humidity. The sampling period was set to three seconds, triggering image acquisition and gas exchange measurements. Different background materials were used to assess the influence of the environment and the camera in both experiments. To separate the plant and background data, static masks were determined. Two datasets were created, which encompass both experiments. One dataset was constructed after averaging over the entire mask to acquire one value per spectral band. These values were then used to calculate a set of vegetation indices. The other dataset used spatial subsampling to retain spatial information. From both datasets, linear models were constructed using ridge regression, which estimated the measured eco-physiological and environmental data. Leaf temperature and vapour pressure deficit based on leaf temperature are the two main eco-physiological characteristics that could be predicted successfully. Stomatal conductance, photosynthesis and transpiration rate show less promising results. We suspect that limited variation, and low spectral resolution and range are the main causes of the inability of the models to extract meaningful predictions. Furthermore, the models that were only trained on background data also showed good predictive performance. This is probably because the main drivers for good performing eco-physiological variables are temperature and incident light intensity. Environmental characteristics that have good performance are photosynthetically active radiation and air temperature. Current hyperspectral sensing technologies are not yet able to uncover most plant dynamic eco-physiological responses when plants are cultivated in stress-free conditions.

Hyperspectral imaging for dynamic thin film interferometry

Dynamic thin film interferometry is a technique used to non-invasively characterize the thickness of thin liquid films that are evolving in both space and time. Recovering the underlying thickness from the captured interferograms, unconditionally and automatically is still an open problem. Here we report a compact setup employing a snapshot hyperspectral camera and the related algorithms for the automated determination of thickness profiles of dynamic thin liquid films. The proposed technique is shown to recover film thickness profiles to within 100 nm of accuracy as compared to those profiles reconstructed through the manual color matching process. Subsequently, we discuss the characteristics and advantages of hyperspectral interferometry including the increased robustness against imaging noise as well as the ability to perform thickness reconstruction without considering the absolute light intensity information.

Irrigation System Using Hyperspectral Data and Machnie Learning Techniques for Smart Agriculture

Water is the main resource for agriculture. Management of water in agricultural field is a challenging process. To manage the water content in the agricultural field, smart irrigation system has been proposed by using fuzzy based decision support system on Hyperspectral Image benchmark dataset. Hyperspectral images are the process of collected and processed the images from electromagnetic spectrum. Recent studies show that hyperspectral images are very accurate in collecting the soil moistures value. Dataset is collected in five-day field of campaign the soil is the type of clayey slit and it is non vegetation. Hyperspectral datasets which consist of range value between 454 to 598 nm. Value is gathered from the 285 hyperspectral snapshot camera recording images with 125 spectral bands with the spectral resolution of 4 nm. Experimental results of this method achieve the accuracy of 0.98. Hence the proposed method reduces the water wastage to an extent.

Enhanced spatial resolution for snapshot hyperspectral imaging of blood perfusion and melanin information within human tissue.

We report a reconstruction method to achieve high spatial resolution for hyperspectral imaging of chromophore features in skin in vivo. The method utilizes an established structure-adaptive normalized convolution algorithm to reconstruct high spatial resolution of hyperspectral images from snapshot low-resolution hyperspectral image sequences captured by a snapshot spectral camera. The reconstructed images at chromophore-sensitive wavebands are used to map the skin features of interest. We demonstrate the method experimentally by mapping the blood perfusion and melanin features (moles) on the facial skin. The method relaxes the constrains of the relatively low spatial resolution in the snapshot hyperspectral camera, making it more usable in imaging applications.

Multimodal Device for Real-Time Monitoring of Skin Oxygen Saturation and Microcirculation Function

The present study introduces a recently developed compact hybrid device for real-time monitoring of skin oxygen saturation and temperature distribution. The prototype involves a snapshot hyperspectral camera, multi-wavelength illuminator, thermal camera, and built-in computer with custom-developed software. To validate this device in-vivo we performed upper arm vascular occlusion on eight healthy volunteers. Palm skin oxygen saturation maps were analyzed in real-time using k-means segmentation algorithm and two-layer optical diffuse model. The prototype system demonstrated a satisfying performance of skin hyperspectral measurements in the spectral range of 507–625 nm. The results confirmed the reliability of the proposed system for in-vivo assessment of skin hemoglobin saturation with oxygen and microcirculation.

The DOM Generation and Precise Radiometric Calibration of a UAV-Mounted Miniature Snapshot Hyperspectral Imager

Hyperspectral remote sensing is used in precision agriculture to remotely and quickly acquire crop phenotype information. This paper describes the generation of a digital orthophoto map (DOM) and radiometric calibration for images taken by a miniaturized snapshot hyperspectral camera mounted on a lightweight unmanned aerial vehicle (UAV). The snapshot camera is a relatively new type of hyperspectral sensor that can acquire an image cube with one spectral and two spatial dimensions at one exposure. The images acquired by the hyperspectral snapshot camera need to be mosaicked together to produce a DOM and radiometrically calibrated before analysis. However, the spatial resolution of hyperspectral cubes is too low to mosaic the images together. Furthermore, there are no systematic radiometric calibration methods or procedures for snapshot hyperspectral images acquired from low-altitude carrier platforms. In this study, we obtained hyperspectral imagery using a snapshot hyperspectral sensor mounted on a UAV. We quantitatively evaluated the radiometric response linearity (RRL) and radiometric response variation (RRV) and proposed a method to correct the RRV effect. We then introduced a method to interpolate position and orientation system (POS) information and generate a DOM with low spatial resolution and a digital elevation model (DEM) using a 3D mesh model built from panchromatic images with high spatial resolution. The relative horizontal geometric precision of the DOM was validated by comparison with a DOM generated from a digital RGB camera. A surface crop model (CSM) was produced from the DEM, and crop height for 48 sampling plots was extracted and compared with the corresponding field-measured crop height to verify the relative precision of the DEM. Finally, we applied two absolute radiometric calibration methods to the generated DOM and verified their accuracy via comparison with spectra measured with an ASD Field Spec Pro spectrometer (Analytical Spectral Devices, Boulder, CO, USA). The DOM had high relative horizontal accuracy, and compared with the digital camera-derived DOM, spatial differences were below 0.05 m (RMSE = 0.035). The determination coefficient for a regression between DEM-derived and field-measured crop height was 0.680. The radiometric precision was 5% for bands between 500 and 945 nm, and the reflectance curve in the infrared spectral region did not decrease as in previous research. The pixel and data sizes for the DOM corresponding to a field area of approximately 85 m × 34 m were small (0.67 m and approximately 13.1 megabytes, respectively), which is convenient for data transmission, preprocessing and analysis. The proposed method for radiometric calibration and DOM generation from hyperspectral cubes can be used to yield hyperspectral imagery products for various applications, particularly precision agriculture.

Use of A Portable Camera for Proximal Soil Sensing with Hyperspectral Image Data

In soil proximal sensing with visible and near-infrared spectroscopy, the currently available hyperspectral snapshot camera technique allows a rapid image data acquisition in a portable mode. This study describes how readings of a hyperspectral camera in the 450–950 nm region could be utilised for estimating soil parameters, which were soil organic carbon (OC), hot-water extractable-C, total nitrogen and clay content; readings were performed in the lab for raw samples without any crushing. As multivariate methods, we used PLSR with full spectra (FS) and also combined with two conceptually different methods of spectral variable selection (CARS, “competitive adaptive reweighted sampling” and IRIV, “iteratively retaining informative variables”). For the accuracy of obtained estimates, it was beneficial to use segmented images instead of image mean spectra, for which we applied a regular decomposing in sub-images all of the same size and k-means clustering. Based on FS-PLSR with image mean spectra, obtained estimates were not useful with RPD values less than 1.50 and R2 values being 0.51 in the best case. With segmented images, improvements were marked for all soil properties; RPD reached values ≥ 1.68 and R2 ≥ 0.66. For all image data and variables, IRIV-PLSR slightly outperformed CARS-PLSR.

GYROCOPTER-BASED REMOTE SENSING PLATFORM

Abstract. In this paper the development of a lightweight and highly modularized airborne sensor platform for remote sensing applications utilizing a gyrocopter as a carrier platform is described. The current sensor configuration consists of a high resolution DSLR camera for VIS-RGB recordings. As a second sensor modality, a snapshot hyperspectral camera was integrated in the aircraft. Moreover a custom-developed thermal imaging system composed of a VIS-PAN camera and a LWIR-camera is used for aerial recordings in the thermal infrared range. Furthermore another custom-developed highly flexible imaging system for high resolution multispectral image acquisition with up to six spectral bands in the VIS-NIR range is presented. The performance of the overall system was tested during several flights with all sensor modalities and the precalculated demands with respect to spatial resolution and reliability were validated. The collected data sets were georeferenced, georectified, orthorectified and then stitched to mosaics.