Spectral Imaging System Research Articles

Interferometer response characterization algorithm for multi-aperture Fabry-Perot imaging spectrometers.

In recent years, the demand for hyperspectral imaging devices has grown significantly, driven by their ability of capturing high-resolution spectral information. Among the several possible optical designs for acquiring hyperspectral images, there is a growing interest in interferometric spectral imaging systems based on division of aperture. These systems have the advantage of capturing snapshot acquisitions while maintaining a compact design. However, they require a careful calibration to operate properly. In this work, we present the interferometer response characterization algorithm (IRCA), a robust three-step procedure designed to characterize the transmittance response of multi-aperture imaging spectrometers based on the interferometry of Fabry-Perot. Additionally, we propose a formulation of the image formation model for such devices suitable to estimate the parameters of interest by considering the model under various regimes of finesse. The proposed algorithm processes the image output obtained from a set of monochromatic light sources and refines the results using nonlinear regression after an ad-hoc initialization. Through experimental analysis conducted on four different prototypes from the Image SPectrometer On Chip (ImSPOC) family, we validate the performance of our approach for characterization. The associated source code for this paper is available from Zenodo (http://dx.doi.org/10.5281/zenodo.7978514).

4-D multiframe co-encoded spectral imaging system

Spectral information can effectively characterise the intrinsic features of a target, while the surface morphology of the object being probed is also a major concern. This paper provides an undersampled spectral 3-D co-imaging scheme. The spectral imaging module of this system shares the optical path with the 3-D imaging module and does not contain a mechanical motion structure, so the scheme is simple in design and stable in structure. A series of experiments have shown that the depth error of this system is within 0.1 mm, the SAM (Spectral Angle Mapper) can reach 4.28 and it has a wide range of applications. It can be applied to non-invasive inspection scenarios such as vegetation monitoring, food safety and heritage conservation.

Multimodal close range hyperspectral imaging combined with multiblock sequential predictive modelling for fresh produce analysis

Multimodal measurements are increasingly becoming common in the domain of spectral sensing and imaging for fresh produce. Often multiple sensors are expected to carry complementary information which allows precise estimation of responses. In this study, a novel case of multimodal hyperspectral imaging is described where two different spectral cameras working in the complementary spectral ranges were integrated into a fully standalone system for spectral imaging for fresh produce analysis. Furthermore, a comparative analysis of different multiblock predictive modelling approaches for fusing data from these two complementary spectral cameras is demonstrated. Both multiblock latent space and multiblock variable selection approaches to identify key variables of interest was examined and compared with the analysis carried out on individual data blocks. Prediction of the soluble solids content in grapes was used to demonstrate the application. The presented approach can increase the applications of multimodal hyperspectral imaging for non-destructive analysis.

Superpixel-based linear reconstruction method for dual-camera compressed hyperspectral imaging system

Compressive spectral imaging systems have received significant research attention in recent years. Among these systems, the dual-camera compressive hyperspectral imaging (DCCHI) design, has the potential to improve imaging resolution while maintaining the snapshot advantage. In this paper, we propose superpixel-based linear reconstruction methods for DCCHI system. Specifically, we assume that the spectral intensity of the superpixels generated from the RGB image varies linearly in the spectral dimension. Instead of addressing an ill-posed problem, we reconstruct hyperspectral images via solving overdetermined equations using the least squares method, resulting in a significant reduction in both computation time and memory usage. To further enhance the reconstruction accuracy, we extend the spectra of superpixels from a 1D linear space into a 3D space with an RGB spectral response matrix. Despite the simplicity of our method, it offers improved reconstruction accuracy of hyperspectral images compared to other state-of-the-art methods. Furthermore, it is more efficient in terms of memory usage and computational time.

Calibration of Acousto-Optic Interaction Geometry Based on the Analysis of AOTF Angular Performance.

Acousto-optic interaction geometry determines the spectral and spatial response of an acousto-optic tunable filter (AOTF). The precise calibration of the acousto-optic interaction geometry of the device is a necessary process before designing and optimizing optical systems. In this paper, we develop a novel calibration method based on the polar angular performance of an AOTF. A commercial AOTF device with unknown geometry parameters was experimentally calibrated. The experimental results show high precision, in some cases falling within 0.01°. In addition, we analyzed the parameter sensitivity and Monte Carlo tolerance of the calibration method. The results of the parameter sensitivity analysis show that the principal refractive index has a large influence on the calibration results, while other factors have little influence. The results of the Monte Carlo tolerance analysis show that the probability of the results falling 0.1° using this method is greater than 99.7%. This work provides an accurate and easy-to-perform method for AOTF crystal calibration and can contribute to the characteristic analysis of AOTFs and the optical design of spectral imaging systems.

Deep image prior plus sparsity prior: toward single-shot full-Stokes spectropolarimetric imaging with a multiple-order retarder

Compressive full-Stokes spectropolarimetric imaging (SPI), integrating passive polarization modulator (PM) into general imaging spectrometer, is powerful enough to capture high-dimensional information via incomplete measurement; a reconstruction algorithm is needed to recover 3D data cube (x, y, and λ) for each Stokes parameter. However, existing PMs usually consist of complex elements and enslave to accurate polarization calibration, current algorithms suffer from poor imaging quality and are subject to noise perturbation. In this work, we present a single multiple-order retarder followed a polarizer to implement passive spectropolarimetric modulation. After building a unified forward imaging model for SPI, we propose a deep image prior plus sparsity prior algorithm for high-quality reconstruction. The method based on untrained network does not need training data or accurate polarization calibration and can simultaneously reconstruct the 3D data cube and achieve self-calibration. Furthermore, we integrate the simplest PM into our miniature snapshot imaging spectrometer to form a single-shot SPI prototype. Both simulations and experiments verify the feasibility and outperformance of our SPI scheme. It provides a paradigm that allows general spectral imaging systems to become passive full-Stokes SPI systems by integrating the simplest PM without changing their intrinsic mechanism.

Polarization Snapshot Imaging Spectrometer for Infrared Range

Infrared imaging spectrometers detect and identify targets by collecting spectral and image information. However, when detecting small temperature differences and dynamic targets, the accuracy of infrared detection is reduced, the traditional scanning structure detection time is longer, the real-time performance is poor and it is easy to introduce motion artifacts. This paper proposes an infrared polarization snapshot spectral imaging system (PSIFTIS) based on a polarizer array, a lens array and a roof-shaped stepped micromirror. Polarized light can solve the problem of small-temperature-difference target recognition by characterizing the surface properties of materials. Lens arrays utilize multi-aperture imaging to achieve snapshot detection of targets. The system can obtain 4D data information, including polarization, in a single measurement cycle. This study completed the overall optical design of a PSIFTIS and an optical simulation experiment using it. Finally, a system prototype was built in the laboratory and a polarization spectrum detection experiment was carried out. The experimental results show that the PSIFTIS could accurately obtain the polarization spectrum information for the target, the spectral resolution reached 7.8 cm−1 and the Stokes measurement error was less than 5%.

A Spectral Encoding Simulator for Broadband Active Illumination and Reconstruction-Based Spectral Measurement

Spectral reflectance or transmittance measurements provide intrinsic information on the material of an object and are widely used in remote sensing, agriculture, diagnostic medicine, etc. Most reconstruction-based spectral reflectance or transmittance measurement methods based on broadband active illumination use narrow-band LEDs or lamps combined with specific filters as spectral encoding light sources. These light sources cannot achieve the designed spectral encoding with a high resolution and accuracy due to their low degree of freedom for adjustment, leading to inaccurate spectral measurements. To address this issue, we designed a spectral encoding simulator for active illumination. The simulator is composed of a prismatic spectral imaging system and a digital micromirror device. The spectral wavelengths and intensity are adjusted by switching the micromirrors. We used it to simulate spectral encodings according to the spectral distribution on micromirrors and solved the DMD patterns corresponding to the spectral encodings with a convex optimization algorithm. To verify the applicability of the simulator for spectral measurements based on active illumination, we used it to numerically simulate existing spectral encodings. We also numerically simulated a high-resolution Gaussian random measurement encoding for compressed sensing and measured the spectral reflectance of one vegetation type and two minerals through numerical simulations. We reconstructed the spectral transmittance of a calibrated filter through an experiment. The results show that the simulator can measure the spectral reflectance or transmittance with a high resolution and accuracy.

Dual camera snapshot high-resolution-hyperspectral imaging system with parallel joint optimization via physics-informed learning.

The hardware architecture of the coded aperture snapshot spectral imaging (CASSI) system is based on a coded mask design, resulting in a poor spatial resolution of the system. Therefore, we consider the use of a physical model of optical imaging and a jointly optimized mathematical model to design a self-supervised framework to solve the high-resolution-hyperspectral imaging problem. In this paper, we design a parallel joint optimization architecture based on a two-camera system. This framework combines the physical model of optical system and a joint optimization mathematical model, which takes full advantage of the spatial detail information provided by the color camera. The system has a strong online self-learning capability for high-resolution-hyperspectral image reconstruction, and gets rid of the dependence of supervised learning neural network methods on training data sets.

Focusing enhancement of terahertz surface plasmon polaritons

Subwavelength-sized metallic structures exhibit extraordinary responses to electromagnetic waves due to their geometry, generating surface plasmon polaritons (SPPs) at the metal–dielectric interface. The development and application of terahertz (THz) science, which is an emerging science, can be advanced by studying SPPs in the THz band. Because the field strength of SPPs gradually weakens in the propagation direction, a coherent phase-matched THz SPP enhancement structure was designed. Here, we demonstrated an asymmetrical circular groove structure. And using this structure, THz SPPs can be excited and can be focused and enhanced under the illumination of linearly polarized THz pulses, generating a subwavelength-sized focal spot. These functions are achieved via phase matching. Furthermore, the original structure was modified by controlling the propagation phase of the THz SPPs and a periodic enhancement structure was designed. Compared to the original structure, the periodic enhancement structure has a stronger focusing effect on THz SPPs. Moreover, we imaged the THz SPPs by using a THz real-time near-field spectral imaging system and verified the feasibility of the designed structure by comparing with the simulation results. The study results are of great significance to the application of THz SPPs and the fabrication of related devices.

FRUIT RIPENESS DETECTION USING DEEP LEARNING

Abstract—The agricultural industry has been facing challenges in traditional and manual visual grading of fruits due to its laborious nature and inconsistent inspection and classification process. To accurately estimate yield and automate harvesting, it is crucial to classify the fruits based on their ripening stages. However, it can be difficult to differentiate between the ripening stages of the same fruit variety due to high similarity in their images during the ripening cycle. To address these challenges, we plan to develop an accurate, fast, and reliable fruit detection sys- tem using deep learning techniques. The modernization of crops offers opportunities for better quality harvests and significant cost savings. Our approach involves adapting the state-of-the- art object detector faster R-CNN, using transfer learning, to detect fruits from images obtained through modalities such as colour (RGB) and Hyper Spectral Imaging System (HSI). Our system’s robustness will enable us to differentiate between fruit varieties and determine the ripening stage of a particular fruit with effectiveness and accuracy. The system will also efficiently segment multiple instances of fruits from an image and accurately grade individual objects. Index Terms—Modernisation, HSI, CNN, RGB

Compact and ultracompact spectral imagers: technology and applications in biomedical imaging.

Spectral imaging, which includes hyperspectral and multispectral imaging, can provide images in numerous wavelength bands within and beyond the visible light spectrum. Emerging technologies that enable compact, portable spectral imaging cameras can facilitate new applications in biomedical imaging. With this review paper, researchers will (1)understand the technological trends of upcoming spectral cameras, (2)understand new specific applications that portable spectral imaging unlocked, and (3)evaluate proper spectral imaging systems for their specific applications. We performed a comprehensive literature review in three databases (Scopus, PubMed, and Web of Science). We included only fully realized systems with definable dimensions. To best accommodate many different definitions of "compact," we included a table of dimensions and weights for systems that met our definition. There is a wide variety of contributions from industry, academic, and hobbyist spaces. A variety of new engineering approaches, such as Fabry-Perot interferometers, spectrally resolved detector array (mosaic array), microelectro-mechanical systems, 3D printing, light-emitting diodes, and smartphones, were used in the construction of compact spectral imaging cameras. In bioimaging applications, these compact devices were used for in vivo and ex vivo diagnosis and surgical settings. Compact and ultracompact spectral imagers are the future of spectral imaging systems. Researchers in the bioimaging fields are building systems that are low-cost, fast in acquisition time, and mobile enough to be handheld.

Investigating the rate of turbidity impact on underwater spectral reflectance detection

Spectral reflectance detection of the targeted object is considered a vital inherent optical property for its potential to provide abundant spectral information, which is crucial in underwater spectral imaging. However, the coarse condition of the underwater environment due to turbidity causes extreme distortions in spectral reflectance detection due to the high absorption and scattering of light. To cope with the effects of light degradation on underwater spectral reflectance detection accuracy, the rate of the impacts of turbidity on spectral reflectance should be examined thoroughly. Therefore, we utilize a stare-type underwater spectral imaging system based on a liquid crystal tunable filter (LCTF) to study the effects of turbidity in underwater spectral imaging of various colored bodies. To examine the accuracy of underwater spectral reflectance detection based on escalating turbidity, the paper models the rate of increase in scattering intensity of the water body. Results show that, based on the non-linear increase in the pixel response of the black and white board, the rapid upsurge in scattering intensity occurs between 400nm to 500nm at different turbidity levels. Additionally, the spectral reconstruction of color bodies relative to the black and white board shows the maximum absolute deviation of 5.3% in spectral reflectance detection accuracy under varying turbidity. While employing underwater spectral imaging, the above findings of optimal band selection can find significant applications to improve the quality of underwater object detection.

A notch-mask and dual-prism system for snapshot spectral imaging

In this paper, we present a notch-mask and dual-prism based system for snapshot spectral imaging. Different from traditional spectrometers that utilize band-pass filters or narrow slits to prevent spectral overlap, losing most of the light emitted from the scene, the main idea of our system is to design a spatial notch mask that only occludes the light of sparsely distributed spectral samplings, and introduce a dual-prism architecture to form a non-dispersive RGB measurement on the detector, resulting in the system’s superiority of high light efficiency and high spatial resolution within the snapshot. Then, we develop a model-based algorithm to reconstruct a high-resolution spectral image from the captured RGB measurement, which is of high tolerance to calibration noise since the spectral information can be directly calculated by a minus operation. Simulated experiments have been conducted to verify the effectiveness of the proposed method, and we also build up a prototype system to further demonstrate the practical application and feasibility of our scheme.

Rapid coded aperture spectrometer based on energy concentration characteristic

In recent years, Coded Aperture Snapshot Spectroscopic Imaging System (CASSI) has attracted more and more attention. However, in the case of undersampling, the effect of the reconstructed image is often unsatisfactory. For spectral imaging system, the slower the imaging speed, the higher the image resolution of hyperspectral images, and vice versa. To solve this problem, in this paper, we design an undersampling rapid coded aperture spectrometer system. Based on the analysis of the internal relationship between coded spectral imaging and single pixel imaging, the order of the discrete cosine transform (DCT) coding patterns is selected by using the energy concentration property of the DCT and the rich spatial information contained in the RGB images. In addition, we built a spectral image enhancement network to improve the quality of the reconstructed images. At the sampling rate of 0.0625, the PSNR, SSIM and SAM of the spectral images reconstructed based on our system are 43.8341, 0.9869 and 2.0862, respectively, the reconstruction effect of this method is significantly superior to other methods, and this method provides an idea to address the contradiction between image resolution and imaging speed to a certain extent, so as to achieve high speed and high resolution hyperspectral imaging.

Abstract P2-21-07: Exploring spatial correlations in Breast invasive Lobular Carcinoma subtypes using a novel CAF multiplex immunofluorescence panel

Abstract Background: Recent advances in immunotherapy led to development of new technologies to identify topographical distributions and correlations in the tumor microenvironment, enabling a better understanding of the tumor immune landscape. Based on the 2019 WHO classification Invasive lobular carcinoma (ILC) has various histological presentations. The pleomorphic variant has been reported to be biologically different to the classical counterpart. Cancer associated fibroblasts (CAFs) secrete cytokines and growth factors aiding in tumor growth. CAFs have been shown to express alpha-smooth muscle actin (α-SMA), Thy1, fibroblast activation protein (FAP), S-100. IHC based studies have shown differences in CAFs subpopulation between ILC and IDC. Here, we explored different CAFs subpopulations between Pleomorphic and Classical ILC using multiplex immunofluorescence (mIF). Study design: Multiplex immunofluorescence (mIF) was performed on formalin-fixed paraffin-embedded (FFPE) tissue sections [(n=6, classic ILC; n=6 pleomorphic ILC)] stained with Opal 7-color Kit using an automated system. Sections were stained consecutively with Pancytokeratin, CD45, A-SMA, FAP, S100, Thy-1. Slides were scanned using the Vectra 3.0 spectral imaging system (PerkinElmer). Five tumor ROIs were examined with Phenochart (Akoya/PerkinElmer) viewer and subsequent images were analyzed to explore spatial distances of CK+ cells to the CAFs. Data was generated using InForm and Phenoptr softwares (Akoya Labs) as a median value. Statistical analysis data correlation was performed using SPSS version 24.0 (IBM Corp) Results: Twelve invasive lobular carcinomas cases were evaluated. All cases with classical features were nuclear grade 2 and all pleomorphic cases (n=6) were nuclear grade 3. Six were Luminal B, five were Luminal A and one case was triple negative invasive lobular carcinoma. We studied nineteen different phenotypes of CAFs in the tumor, stroma, and overall tissue compartments. We found statistically significant differences (p-value< 0.05) between the distances of CAFs from the tumor cells in classic and pleomorphic ILC in three phenotypes. The A-SMA+, A-SMA+/S100+ phenotypes were closer to the tumor cells in classic subtype and the S100 only phenotype was closer to the tumor cells in pleomorphic carcinomas. In addition, there were two other phenotypes namely A-SMA+/Thy-1+(closer to tumor cells in classic ILC) and FAP+/S-100+ (closer to tumor cells in pleomorphic ILC), which showed a trend of significance (Table 1) Discussion: TME studies in invasive breast lobular carcinoma have been primarily based on chromogenic IHC. Multiplex immunofluorescence quantifies cell phenotype’s densities and positions in different tissue compartments (tumor vs stroma vs total tissue). Knowing the proximity of an individual TME cells to the tumor cells, aids in pinpointing possible therapeutic targets. Fibroblast dysregulation in cancers, enhances their pro-tumorigenic and anti-tumorigenic potential. Due to their heterogeneity, they can express a wide array of markers. Identifying the full possibility of CAF subsets is one of the keys to discovering actionable targets. Our results showed Alpha- SMA positive, Alpha-SMA/S-100 positive, Alpha SMA/Thy-1 positive CAFs in closer proximity to the classic ILC tumor cells as compared to pleomorphic ILC. Additionally, S-100 positive and FAP/S-100 positive CAFs showed a closer proximity to tumor cells in the pleomorphic subtype. Based on these results we hypothesize that pleomorphic ILCs are closely associated with pro-tumorigenic CAFs as compared to classic ILCs. Larger datasets are needed to confirm this. In conclusion we utilized an objective approach to quantify, phenotype and spatially correlate each cell in the tumor microenvironment. This has helped identifying CAFs subsets that differs in the spatial correlation to tumor cells in ILC subtypes, which can be potential actionable targets. Table 1 Phenotypes of Cancer associated fibroblasts and their statistical correlation between classic vs pleomorphic invasive Lobular carcinomas. Citation Format: Harsh Batra, Renganayaki K. Pandurengan, Heladio P. Ibarguen, Salome A McAllen, Qingqing Ding, Aysegul Sahin, Ignacio Wistuba, Edwin Roger Parra, Maria Gabriela Raso. Exploring spatial correlations in Breast invasive Lobular Carcinoma subtypes using a novel CAF multiplex immunofluorescence panel [abstract]. In: Proceedings of the 2022 San Antonio Breast Cancer Symposium; 2022 Dec 6-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2023;83(5 Suppl):Abstract nr P2-21-07.

The Development of Snapshot Multispectral Imaging Technology Based on Artificial Compound Eyes

In the present study, the advantages of multispectral imaging over hyperspectral imaging in real-time spectral imaging are briefly analyzed, and the advantages and disadvantages of snapshot spectral imaging and other spectral imaging technologies are briefly described. The technical characteristics of artificial compound eyes and multi-aperture imaging and the research significance of snapshot artificial compound eye multispectral imaging are also introduced. The classification and working principle of the snapshot artificial compound eye multispectral imaging system are briefly described. According to the realization method of the optical imaging system, the ACE snapshot multi-aperture multispectral imaging system is divided into plane and curved types. In the planar compound eye spectral imaging system, the technical progress of the multispectral imaging system based on the thin observation module by bound optics (TOMBO) architecture and the multispectral imaging system based on the linear variable spectral filter are introduced. At the same time, three curved multispectral imaging systems are introduced. Snapshot artificial compound eye multispectral imaging technology is also briefly analyzed and compared. The research results are helpful to comprehensively understand the research status of snapshot multispectral multi-aperture imaging technology based on artificial compound eyes and to lay the foundation for improving its comprehensive performance even further.

Recurrent Neural Networks for Snapshot Compressive Imaging.

Conventional high-speed and spectral imaging systems are expensive and they usually consume a significant amount of memory and bandwidth to save and transmit the high-dimensional data. By contrast, snapshot compressive imaging (SCI), where multiple sequential frames are coded by different masks and then summed to a single measurement, is a promising idea to use a 2-dimensional camera to capture 3-dimensional scenes. In this paper, we consider the reconstruction problem in SCI, i.e., recovering a series of scenes from a compressed measurement. Specifically, the measurement and modulation masks are fed into our proposed network, dubbed BIdirectional Recurrent Neural networks with Adversarial Training (BIRNAT) to reconstruct the desired frames. BIRNAT employs a deep convolutional neural network with residual blocks and self-attention to reconstruct the first frame, based on which a bidirectional recurrent neural network is utilized to sequentially reconstruct the following frames. Moreover, we build an extended BIRNAT-color algorithm for color videos aiming at joint reconstruction and demosaicing. Extensive results on both video and spectral, simulation and real data from three SCI cameras demonstrate the superior performance of BIRNAT.

Development of an accurate low cost NDVI imaging system for assessing plant health

BackgroundSpectral imaging is a key method for high throughput phenotyping that can be related to a large variety of biological parameters. The Normalised Difference Vegetation Index (NDVI), uses specific wavelengths to compare crop health and performance. Increasing the accessibility of spectral imaging systems through the development of small, low cost, and easy to use platforms will generalise its use for precision agriculture. We describe a method for using a dual camera system connected to a Raspberry Pi to produce NDVI imagery, referred to as NDVIpi. Spectral reference targets were used to calibrate images into values of reflectance, that are then used to calculated NDVI with improved accuracy compared with systems that use single references/standards.ResultsNDVIpi imagery showed strong performance against standard spectrometry, as an accurate measurement of leaf NDVI. The NDVIpi was also compared to a relatively more expensive commercial camera (Micasense RedEdge), with both cameras having a comparable performance in measuring NDVI. There were differences between the NDVI values of the NDVIpi and the RedEdge, which could be attributed to the measurement of different wavelengths for use in the NDVI calculation by each camera. Subsequently, the wavelengths used by the NDVIpi show greater sensitivity to changes in chlorophyll content than the RedEdge.ConclusionWe present a methodology for a Raspberry Pi based NDVI imaging system that utilizes low cost, off-the-shelf components, and a robust multi-reference calibration protocols that provides accurate NDVI measurements. When compared with a commercial system, comparable NDVI values were obtained, despite the fact that our system was a fraction of the cost. Our results also highlight the importance of the choice of red wavelengths in the calculation of NDVI, which resulted in differences in sensitivity between camera systems.

Study of multispectral polarization imaging in sea fog environment

Marine exploration has become a popular field of concern and research all over the world. The impact of sea fog on ocean exploration is very great, and how to carry out accurate identification of targets in the sea fog environment is a problem we urgently need to solve. In this paper, we simulated and analyzed the particle distribution characteristics of the sea fog layer by using the principle of Mie scattering, designed a spectral polarization imaging system by using Liquid Crystal Variable Retarder (LCVR) and Liquid Crystal Tunable Filter (LCTF) according to the principle of spectral spectroscopy and polarization imaging, conducted calibration experiments on liquid crystal tunable filter, and carried out experiments on visibility, wavelength and imaging distance that affect the experimental results of polarization imaging of sea fog environment. The experimental results show that the polarization decreases with the increase of imaging distance; in the typical visibility (5 km for light fog, 0.5 km for medium fog and 0.05 km for dense fog), the higher the visibility, the higher the polarization; for the typical wavelengths of visible light (450 nm, 530 nm and 670 nm), the polarization increase with the increase of wavelength.