Snapshot Hyperspectral Camera Research Articles

Seeing the light: Hyperspectral retinal imaging as a surrogate marker for early stage Alzheimer's disease

Aims/Purpose: Alzheimer's disease (AD), the leading cause of dementia, affects millions, underscoring the need for early diagnostic methods. The retina, which can exhibit AD‐related pathological processes, offers a promising diagnostic site. This study aims to utilize Hyperspectral Retinal Imaging (HSRI) to differentiate between positive (+) and negative (‐) PET‐amyloid beta (Aβ) status. Within this cohort, a subset is Aβ+ while remaining cognitively healthy, indicating preclinical amyloid accumulation. Unlike most research targeting later symptomatic stages, our approach could facilitate interventions before clinically apparent neurodegeneration occurs.Methods: Data was used from 37 Aβ‐ and 25 Aβ+ individuals of which 8 with mild cognitive impairment (MCI). HSRI were captured using an easy‐to‐use hyperspectral snapshot camera, covering a spectral range of 460–630 nm. Images were taken from primary, inferior, and superior views of the retina, with further regions of interest (ROI) defined in the primary view. After preprocessing, these spectra were used to train machine learning classifiers [1].Results: The best classifier performance was observed using XGBoost with the superior spectra (AUC = 0.72), and using a Linear Discriminant Analysis classifier in superior quadrants (S1 AUC = 0.74, S2 AUC = 0.68). Lower performance in other ROI underscored the variability in amyloid deposition across the retina, correlating with previous findings that retinal Aβ deposits in AD patients are frequently concentrated in the superior quadrants.Conclusions: We were able to achieve similar performance of previous research in AD patients [1], but in a preclinical cohort of cognitively normal participants up to MCI patients. Consistent with the literature, the superior retinal regions seem most discriminative for Aβ status. Further exploration of the specificity of the HSRI signal will pave the way for early AD detection, potentially improving outcomes through earlier therapeutic intervention.Reference S. Lemmens et al., ‘Combination of snapshot hyperspectral retinal imaging and optical coherence tomography to identify Alzheimer's disease patients’, Alzheimer's Research & Therapy, vol. 12, no. 1, p. 144, Nov. 2020, doi: 10.1186/s13195‐020‐00715‐1.

High-resolution hyperspectral video imaging using a hexagonal camera array.

Retrieving the reflectance spectrum from objects is an essential task for many classification and detection problems, since many materials and processes have a unique spectral behavior. In many cases, it is highly desirable to capture hyperspectral images due to the high spectral flexibility. Often, it is even necessary to capture hyperspectral videos or at least to be able to record a hyperspectral image at once, also called snapshot hyperspectral imaging, to avoid spectral smearing. For this task, a high-resolution snapshot hyperspectral camera array using a hexagonal shape is introduced. The hexagonal array for hyperspectral imaging uses off-the-shelf hardware, which enables high flexibility regarding employed cameras, lenses, and filters. Hence, the spectral range can be easily varied by mounting a different set of filters. Moreover, the concept of using off-the-shelf hardware enables low prices in comparison to other approaches with highly specialized hardware. Since classical industrial cameras are used in this hyperspectral camera array, the spatial and temporal resolution is very high, while recording 37 hyperspectral channels in the range from 400 to 760nm in 10nm steps. As the cameras are at different spatial positions, a registration process is required for near-field imaging, which maps the peripheral camera views to the center view. It is shown that this combination using a hyperspectral camera array and the corresponding image registration pipeline is superior in comparison to other popular snapshot approaches. For this evaluation, a synthetic hyperspectral database is rendered. On the synthetic data, the novel approach, to our knowledge, outperforms its best competitor by more than 3dB in reconstruction quality. This synthetic data is also used to show the superiority of the hexagonal shape in comparison to an orthogonal-spaced one. Moreover, a real-world high-resolution hyperspectral video database with 10 scenes is provided for further research in other applications.

Rapid Detection of Single Bacteria Using Filter-Array-Based Hyperspectral Imaging Technology.

Rapid and accurate detection of bacterial pathogens is crucial for preventing widespread public health crises, particularly in the food industry. Traditional methods are often slow and require extensive labeling, which hampers timely responses to potential threats. In response, we introduce a groundbreaking approach using filter-array-based hyperspectral imaging technology, enhanced by a super-resolution demosaicking technique. This innovative technology streamlines the detection process and significantly enhances the resolution of mosaic hyperspectral imaging. By utilizing a snapshot hyperspectral camera with a 15 ms integration time, it facilitates the identification of bacteria at the single-cell level without requiring chemical labels. The integration of a 3D convolutional neural network optimizes the recognition of pathogenic bacteria, achieving an impressive accuracy of 91.7%. Our approach dramatically improves the efficiency and effectiveness of bacterial detection, providing a promising solution for critical applications in public health and the food industry.

Synchronising an IMX219 image sensor and AS7265x spectral sensor to make a novel low-cost spectral camera

A low-cost novel spectral camera able to be used for near infrared spectroscopy was made by using a Jetson Nano to synchronize a Sony IMX219 NOIR autofocus image sensor, an AMS AS7265x 18-channel spectral sensor and Osram SFH 4737 broadband infrared LED’s. Synchronizing an image sensor and spectral sensor augments a standard RGB image with light spectrum information; capturing the light distribution information normally lost in RGB image capture. Sutherland et al. [1] used this novel spectral camera to examine the dorsal surface of juvenile lobsters as a possible pre-moult detector. Having the image and spectrum in combination allowed the incomplete and unmineralized post-moult dorsal surface to be characterized with 86.7% accuracy for the first time. A proposed application for the spectral camera is to omit the local SFH 4737 light source and use the camera in daylight, effectively making a low-cost substitute hyperspectral snapshot camera. In this configuration the camera may have application for low-cost drone deployment for small scale agriculture.

Dual-camera laparoscopic imaging with super-resolution reconstruction for intraoperative hyperspectral image guidance.

Laparoscopic and robotic surgery, as one type of minimally invasive surgery (MIS), has gained popularity due to the improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery. Hyperspectral imaging (HSI) is an emerging medical imaging modality, which has proved useful for intraoperative image guidance. Snapshot hyperspectral cameras are ideal for intraoperative laparoscopic imaging because of their compact size and light weight, but low spatial resolution can be a limitation. In this work, we developed a dual-camera laparoscopic imaging system that consists of a high-resolution color camera and a snapshot hyperspectral camera, and we employed super-resolution reconstruction to fuse the images from both cameras to generate high-resolution hyperspectral images. The experimental results show that our method can significantly improve the resolution of hyperspectral images without compromising the image quality or spectral signatures. The proposed super-resolution reconstruction method is promising to promote the employment of high-speed hyperspectral imaging in laparoscopic surgery.

Compact snapshot hyperspectral camera for ophthalmology

Hyperspectral imaging is an emerging technique that allows measurement of spectral absorption at each point of a scene, thus offering the capability to identify and characterize important biomarkers for clinical practice and therapeutic research, as well as enhancing image identification of important structures. So far, few hyperspectral cameras have been used for retinal scanning because of the need to acquire the image in a fraction of a second. Here, and to the best of our knowledge, we present a novel concept of a snapshot hyperspectral camera suited for retinal imaging. We demonstrate the technique by presenting the optical density spectrum of a healthy patient’s retina in the 450–700 nm range, together with the spectral response of several retinal features.

A High-Resolution Hyperspectral Imaging System for the Retina.

In this study, we developed an imaging system that can acquire and produce high-resolution hyperspectral images of the retina. Our system combines the view from a high-resolution RGB camera and a snapshot hyperspectral camera together. The method is fast and can be constructed into a compact imaging device. We tested our system by imaging a calibrated color chart, biological tissues ex vivo, and a phantom of the human retina. By using image pansharpening methods, we were able to produce a high-resolution hyperspectral image. The images from the hyperspectral camera alone have a spatial resolution of 0.2 mm/pixel, whereas the pansharpened images have a spatial resolution of 0.1 mm/pixel, a 2x increase in spatial resolution. Our method has the potential to capture images of the retina rapidly. Our method preserves both the spatial and spectral fidelity, as shown by comparing the original hyperspectral images with the pansharpened images. The high-resolution hyperspectral imaging device can have a variety of applications in retina examinations.

An open-source machine-learning application for predicting pixel-to-pixel NDVI regression from RGB calibrated images

The Normalized Difference Vegetation Index (NDVI) is the most common index to measure vegetation in agriculture and create classified prescription maps used for several purposes. It is obtained as the ratio between information from the visible and near infrared spectral bands. The calculation of NDVI often uses very expensive hyperspectral and multispectral optical sensors. This study aims to develop an efficient model to extract NDVI data from RGB images. During the work were acquired images, of different plant species at different vegetation status, through a snapshot hyperspectral camera (Specim IQ) featuring natively superimposed RGB sensor whose images were calibrated a posteriori (sRGB). NDVI was predicted from sRGB images using a Shallow-regressive neural network that performs a pixel-pixel regression. The model has been then tested on around 1000 drone images acquired by a 6x Sentera sensor to test the efficiency of the applied model. The model has been used to estimate the numerical value of NDVI and the classes produced by a clustering method (k-means). The results of this work showed that the calibrated images had a very high correlation with the NDVI in validation (r = 0.91), maintaining good performances (r = 0.71) when applied to a set of data acquired with a different non-co-registered sensor. Therefore, for the first time this work has shown that the health of vegetation through the NDVI could be calculated using unexpensive RGB device adopting a pixel-pixel regression AI approach. The approach shows its importance especially for small-sized farms where profit would not allow for the budget to access multispectral cameras and heavy carriers (i.e., UAV).

Aperture-encoded snapshot hyperspectral imaging with a lensless camera

We present a lensless snapshot hyperspectral camera that is capable of hyperspectral imaging over a broad spectrum using a compact and low-cost hardware configuration. We leverage the multiplexing capability of a lensless camera, a novel type of computational imaging device that replaces the lens with a thin mask. Our device utilizes a linear variable filter and a phase mask to encode spectral information onto a monochromatic image sensor, enabling recovery of hyperspectral image stacks from a single measurement by utilizing spectral information encoded in different parts of the 2D point spread function. We perform spectral calibration using a reference color chart and verify the prototype device’s spectral and spatial resolution, as well as its imaging field of view. We report on the design and construction of the device, the image reconstruction algorithm, and spectral calibration methods and present hyperspectral images ranging from 410 to 800 nm obtained with our prototype device.

Automated Polarized Hyperspectral Imaging (PHSI) for ex-vivo and in-vivo Tissue Assessment.

Polarized light interactions with biological tissues can reveal information regarding tissue structure, while spectral characteristics are closely related to tissue composition. An integration of both modalities in a compact system could better assist tissue assessment. This study aims to develop a polarized hyperspectral imaging (PHSI) system that fulfills both linearly and circularly polarized hyperspectral imaging for in vivo and ex vivo applications. The system is comprised of a white LED, two linear polarizers, two liquid crystal variable retarders (LCVRs), and a hyperspectral snapshot camera. The system was calibrated to compute the full Stokes polarimetry. For tissue differentiation, fresh ex vivo mouse tissue specimens from kidney, liver, spleen, muscle, lung, and salivary gland of mice were imaged. The spectra of three features, named degree of polarization (DOP), degree of linear polarization (DOLP), and degree of circular polarization (DOCP), were generated. A k-nearest neighbor (k-NN) classifier was trained with multi-class spectra and 5-fold cross validation. It was found that DOP better differentiates tissue with an average accuracy of 0.87. Additionally, support vector machine (SVM) classifiers were trained to differentiate between each two of the organs, and it was determined that DOLP better identified kidney, liver, and spleen, whereas DOCP better identified muscle and lung tissues. Then, the setup was employed to image in vivo human fingers with and without a blood occlusion to qualitatively estimate oxygen saturation. Preliminary results demonstrate that both DOLP and DOCP reveal a distinction of oxygen saturation states. These results demonstrate the feasibility of the PHSI system for distinguishing between optical properties of tissues, which has the potential to reveal disease-related information for diverse medical applications.

Design and realization of a miniaturized high resolution computed tomography imaging spectrometer

The computed tomography imaging spectrometer (CTIS) is a relatively unknown snapshot hyperspectral camera. It utilizes computational imaging approaches to gain the hyperspectral image from a spatio-spectral smeared sensor image. We present a strongly miniaturized system with a dimension of only 36 × 40.5 × 52.8 mm and a diagonal field of view of 29°. We achieve this using a Galilean beam expander and a combination of off-the-shelf lenses, a highly aspherical imaging system from a commercial smartphone, and a 13 MP monochrome smartphone image sensor. The reconstructed hyperspectral image has a spatial resolution of 400 × 300 pixel with 39 spectral channels.

Design and realization of a miniaturized high resolution computed tomography imaging spectrometer

The computed tomography imaging spectrometer (CTIS) is a relatively unknown snapshot hyperspectral camera. It utilizes computational imaging approaches to gain the hyperspectral image from a spatiospectral smeared sensor image. We present a strongly miniaturized system with a dimension of only 36mm x 40.5mm x 52.8mm and a diagonal field of view of 29°. We achieve this using a Galilean beam expander and a combination of off-the-shelf lenses, a highly aspherical imaging system from a commercial smartphone and a 13 MP monochrome smartphone image sensor. The reconstructed hyperspectral image has a spatial resolution of 400 x 300 pixel with 39 spectral channels.

Detection of cutaneous oxygen saturation using a novel snapshot hyperspectral camera: a feasibility study.

Tissue necrosis, a consequence of inadequate tissue oxygenation, is a common post-operative complication. As current surgical assessments are often limited to visual and tactile feedback, additional techniques that can aid in the interrogation of tissue viability are needed to improve patient outcomes. In this bi-institutional pilot study, the performance of a novel snapshot hyperspectral imaging camera to detect superficial cutaneous oxygen saturation (StO2) was evaluated. Healthy human volunteers were recruited at two participating centers. Cutaneous StO2 of the forearm was determined by a snapshot hyperspectral camera on two separate study days during occlusion-reperfusion of the brachial artery and after induction of local vasodilation. To calculate the blood StO2 at each pixel in the multispectral image, spectra were selected, and fitting was performed over wavelengths ranging from 470 to 950 nm. Quantitative detection of physiological changes in cutaneous StO2 levels was feasible in all sixteen volunteers. A significant (P<0.001) decrease in cutaneous StO2 levels from 78.3% (SD: 15.3) at baseline to 60.6% (SD: 19.8) at the end of occlusion phase was observed, although StO2 levels returned to baseline after five minutes. Mean cutaneous StO2 values were similar in the same subjects on separate study days (Pearson R2: 0.92 and 0.77, respectively) at both centers. Local vasodilation did not yield significant changes in cutaneous StO2 values. This pilot study demonstrated the feasibility of a snapshot hyperspectral camera for detecting quantitative physiological changes in cutaneous StO2 in normal human volunteers, and serves as a precursor for further validation in perioperative studies.

3D-printed miniature spectrometer for the visible range with a 100 × 100 μm&lt;sup&gt;2&lt;/sup&gt; footprint

The miniaturisation of spectroscopic measurement devices opens novel information channels for size critical applications such as endoscopy or consumer electronics. Computational spectrometers in the micrometre size range have been demonstrated, however, these are calibration sensitive and based on complex reconstruction algorithms. Herein we present an angle-insensitive 3D-printed miniature spectrometer with a direct separated spatial-spectral response. The spectrometer was fabricated via two-photon direct laser writing combined with a super-fine inkjet process. It has a volume of less than 100 × 100 × 300 μm3. Its tailored and chirped high-frequency grating enables strongly dispersive behaviour. The miniature spectrometer features a wavelength range of 200 nm in the visible range from 490 nm to 690 nm. It has a spectral resolution of 9.2 ± 1.1 nm at 532 nm and 17.8 ± 1.7 nm at a wavelength of 633 nm. Printing this spectrometer directly onto camera sensors is feasible and can be replicated for use as a macro-pixel of a snapshot hyperspectral camera.

Combination of snapshot hyperspectral retinal imaging and optical coherence tomography to identify Alzheimer\u2019s disease patients

IntroductionThe eye offers potential for the diagnosis of Alzheimer’s disease (AD) with retinal imaging techniques being explored to quantify amyloid accumulation and aspects of neurodegeneration. To assess these changes, this proof-of-concept study combined hyperspectral imaging and optical coherence tomography to build a classification model to differentiate between AD patients and controls.MethodsIn a memory clinic setting, patients with a diagnosis of clinically probable AD (n = 10) or biomarker-proven AD (n = 7) and controls (n = 22) underwent non-invasive retinal imaging with an easy-to-use hyperspectral snapshot camera that collects information from 16 spectral bands (460–620 nm, 10-nm bandwidth) in one capture. The individuals were also imaged using optical coherence tomography for assessing retinal nerve fiber layer thickness (RNFL). Dedicated image preprocessing analysis was followed by machine learning to discriminate between both groups.ResultsHyperspectral data and retinal nerve fiber layer thickness data were used in a linear discriminant classification model to discriminate between AD patients and controls. Nested leave-one-out cross-validation resulted in a fair accuracy, providing an area under the receiver operating characteristic curve of 0.74 (95% confidence interval [0.60–0.89]). Inner loop results showed that the inclusion of the RNFL features resulted in an improvement of the area under the receiver operating characteristic curve: for the most informative region assessed, the average area under the receiver operating characteristic curve was 0.70 (95% confidence interval [0.55, 0.86]) and 0.79 (95% confidence interval [0.65, 0.93]), respectively. The robust statistics used in this study reduces the risk of overfitting and partly compensates for the limited sample size.ConclusionsThis study in a memory-clinic-based cohort supports the potential of hyperspectral imaging and suggests an added value of combining retinal imaging modalities. Standardization and longitudinal data on fully amyloid-phenotyped cohorts are required to elucidate the relationship between retinal structure and cognitive function and to evaluate the robustness of the classification model.

Limitations of snapshot hyperspectral cameras to monitor plant response dynamics in stress-free conditions

Plants’ dynamic eco-physiological responses are vital to their productivity in continuously fluctuating conditions, such as those in agricultural fields. However, it is currently still very difficult to capture these responses at the field scale for phenotyping purposes. Advanced hyperspectral imaging tools are increasingly used in phenotyping, and have been applied to detect changes in plants in response to a specific treatment, phenological state or monitor its growth and development. Phenotyping has to evolve towards capturing dynamic behaviour under more subtle fluctuations in environmental conditions, without the presence of clear treatments or stresses. Therefore, we investigated the potential of hyperspectral imaging to capture dynamic behaviour of plants in stress-free conditions at a temporal resolution of seconds. Two growth chamber experiments were set up, in which strawberry plants and four different background materials, serving as controls, were monitored by a snapshot hyperspectral camera in variable conditions of light, temperature and relative humidity. The sampling period was set to three seconds, triggering image acquisition and gas exchange measurements. Different background materials were used to assess the influence of the environment and the camera in both experiments. To separate the plant and background data, static masks were determined. Two datasets were created, which encompass both experiments. One dataset was constructed after averaging over the entire mask to acquire one value per spectral band. These values were then used to calculate a set of vegetation indices. The other dataset used spatial subsampling to retain spatial information. From both datasets, linear models were constructed using ridge regression, which estimated the measured eco-physiological and environmental data. Leaf temperature and vapour pressure deficit based on leaf temperature are the two main eco-physiological characteristics that could be predicted successfully. Stomatal conductance, photosynthesis and transpiration rate show less promising results. We suspect that limited variation, and low spectral resolution and range are the main causes of the inability of the models to extract meaningful predictions. Furthermore, the models that were only trained on background data also showed good predictive performance. This is probably because the main drivers for good performing eco-physiological variables are temperature and incident light intensity. Environmental characteristics that have good performance are photosynthetically active radiation and air temperature. Current hyperspectral sensing technologies are not yet able to uncover most plant dynamic eco-physiological responses when plants are cultivated in stress-free conditions.

Hyperspectral imaging for dynamic thin film interferometry

Dynamic thin film interferometry is a technique used to non-invasively characterize the thickness of thin liquid films that are evolving in both space and time. Recovering the underlying thickness from the captured interferograms, unconditionally and automatically is still an open problem. Here we report a compact setup employing a snapshot hyperspectral camera and the related algorithms for the automated determination of thickness profiles of dynamic thin liquid films. The proposed technique is shown to recover film thickness profiles to within 100 nm of accuracy as compared to those profiles reconstructed through the manual color matching process. Subsequently, we discuss the characteristics and advantages of hyperspectral interferometry including the increased robustness against imaging noise as well as the ability to perform thickness reconstruction without considering the absolute light intensity information.

Irrigation System Using Hyperspectral Data and Machnie Learning Techniques for Smart Agriculture

Water is the main resource for agriculture. Management of water in agricultural field is a challenging process. To manage the water content in the agricultural field, smart irrigation system has been proposed by using fuzzy based decision support system on Hyperspectral Image benchmark dataset. Hyperspectral images are the process of collected and processed the images from electromagnetic spectrum. Recent studies show that hyperspectral images are very accurate in collecting the soil moistures value. Dataset is collected in five-day field of campaign the soil is the type of clayey slit and it is non vegetation. Hyperspectral datasets which consist of range value between 454 to 598 nm. Value is gathered from the 285 hyperspectral snapshot camera recording images with 125 spectral bands with the spectral resolution of 4 nm. Experimental results of this method achieve the accuracy of 0.98. Hence the proposed method reduces the water wastage to an extent.

Enhanced spatial resolution for snapshot hyperspectral imaging of blood perfusion and melanin information within human tissue.

We report a reconstruction method to achieve high spatial resolution for hyperspectral imaging of chromophore features in skin in vivo. The method utilizes an established structure-adaptive normalized convolution algorithm to reconstruct high spatial resolution of hyperspectral images from snapshot low-resolution hyperspectral image sequences captured by a snapshot spectral camera. The reconstructed images at chromophore-sensitive wavebands are used to map the skin features of interest. We demonstrate the method experimentally by mapping the blood perfusion and melanin features (moles) on the facial skin. The method relaxes the constrains of the relatively low spatial resolution in the snapshot hyperspectral camera, making it more usable in imaging applications.

Multimodal Device for Real-Time Monitoring of Skin Oxygen Saturation and Microcirculation Function

The present study introduces a recently developed compact hybrid device for real-time monitoring of skin oxygen saturation and temperature distribution. The prototype involves a snapshot hyperspectral camera, multi-wavelength illuminator, thermal camera, and built-in computer with custom-developed software. To validate this device in-vivo we performed upper arm vascular occlusion on eight healthy volunteers. Palm skin oxygen saturation maps were analyzed in real-time using k-means segmentation algorithm and two-layer optical diffuse model. The prototype system demonstrated a satisfying performance of skin hyperspectral measurements in the spectral range of 507–625 nm. The results confirmed the reliability of the proposed system for in-vivo assessment of skin hemoglobin saturation with oxygen and microcirculation.