Spectral Imaging System Research Articles

Research on the Processing of Image and Spectral Information in an Infrared Polarization Snapshot Spectral Imaging System

In order to solve the problems of a low target recognition rate and poor real-time performance brought about by conventional infrared imaging spectral detection technology under complex background conditions or in the detection of targets of weak radiation or long distance, a kind of infrared polarization snapshot spectral imaging system (PSIFTIS) and a spectrum information processing method based on micro-optical devices are proposed in this paper, where the synchronous acquisition of polarization spectrum information is realized through the spatial modulation of phase with a rooftop-shaped multi-stage micro-mirror and the modulation of the polarization state of light with a micro-nanowire array. For the polarization interference image information obtained, the infrared polarization spectrum decoupling is realized by image segmentation, optical path difference matching, and image registration methods, the infrared polarization spectrum reconstruction is realized by Fourier transform spectral demodulation, and the infrared polarization image fusion is realized by decomposing and reconstructing the high- and low-frequency components of the polarization image based on the Haar wavelet transform. The maximum spectral peak wavenumber error of the four polarization channels of the polarization spectrum reconstruction is less than 2 cm−1, and the polarization angle error is within 1°. Ultimately, compared with the unprocessed polarization image unit, the peak signal-to-noise ratio is improved by 45.67%, the average gradient is improved by 8.03%, and the information entropy is improved by 56.98%.

Optimization of a real-time LED based spectral imaging system.

We are designing a real-time spectral imaging system using micro-LEDs and a high-speed micro-camera for potential endoscopic applications. Currently, gastrointestinal (GI) endoscopic imaging has largely been limited to white light imaging (WLI), while other endoscopic approaches have seen advancements in imaging techniques including fluorescence imaging, narrow-band imaging, and stereoscopic visualization. To further advance GI endoscopic imaging, we are working towards a high-speed spectral imaging system that can be implemented with a chip-on-tip design for a flexible endoscope. Hyperspectral imaging has potential applications in a variety of imaging procedures with its ability to discern spectral footprints of the imaging field in a series of two-dimensional images at different wavelengths. For investigating the feasibility of a real-time LED-based hyperspectral imaging system for endoscopic applications we designed and developed a large-scale prototype using through-hole LEDs, which is further analyzed as we design our future system. For high quality imaging, the LED array must be designed with the specific illumination patterns and intensities of each LED considered. We present our work in optimizing our current LED array through optical simulations performed in silico. Damped least squares and orthogonal descent algorithms are implemented to maximize irradiance power and improve illumination homogeneity at several working distances by adjusting radial distances of each LED from the camera. With our prototype LED-based hyperspectral imaging system, the simulation and optimization approach achieved an average increase of 8.36 ± 8.79% in irradiance and a 4.3% decrease in standard deviation at multiple working distances and field-of-views for each LED, as compared to the original design, leading to improved image quality and maintained acquisition speeds. This work highlights the value of in silico optical simulation and provides a framework for improved optical system design and will inform design decisions in future works.

Supervised-unsupervised combined transformer for spectral compressive imaging reconstruction

To solve the low spatial and/or temporal resolution problem which the conventional hyperspectral cameras often suffer from, spectral compressive imaging systems (SCI) have attracted more attention recently. Recovering a hyperspectral image (HSI) from its corresponding 2D coded image is an ill-posed inverse problem, and learning accurate prior from HSI and 2D coded image is essential to solve this inverse problem. Existing methods only use supervised networks that focus on learning generalized prior from training datasets, or only use unsupervised networks that focus on learning specific prior from 2D coded image, resulting in the inability to learn both generalized and specific priors. Also, when learning the priors, existing methods cannot simultaneously give consideration to both global and local scales, as well as both spatial and spectral dimensions. To cope with this problem, in this paper, we propose a Supervised-Unsupervised Combined Transformer Network (SUCTNet) composed by a supervised Spatio-spectral Transformer network (SSTNet) and an Unsupervised Multi-level Feature Refinement network (UMFRNet). Specifically, we first develop the SSTNet to learn generalized prior and obtain a preliminary HSI. In SSTNet, the proposed spatial encoding and spectral decoding network architecture enables it to simultaneously consider both spatial and spectral dimensions, and a proposed Global and Local Multi head Self Attention block (GL-MSA) enables it simultaneously to consider both global and local scales. Then, the preliminary HSI is fed into the proposed UMFRNet to learn specific prior and obtain the target HSI. In UMFRNet, a proposed multi-level feature refinement mechanism and the physical imaging model of SCI are used to improve reconstruction accuracy and generalization performance. Extensive experiments show that our method significantly outperforms state-of-the-art (SOTA) methods on simulated and real datasets. Codes will be available at https://github.com/Vzhouhan/SUCTNet.

Anti-laser interference methods for compressive spectral imaging based on grayscale coded aperture

Coded aperture snapshot spectral imaging (CASSI) captures three-dimensional (3D) spectral images by one or several snapshots two-dimensional (2D) measurements. At present, in practical application of spectral imaging systems, artificial interference such as laser irradiation is wide-spread. When CASSI system is irradiated by intense laser, optoelectronic image sensor will inevitably be dazzled, leading to the loss of compressive sampling information, thereby reducing the reconstruction accuracy of spectral images. This paper establishes the imaging model of CASSI system under laser interference and proposes a grayscale coding design method. Through the design of a reasonable and feasible feedback model, the grayscale coded aperture is calculated according to the measurements of CASSI system in real time feedback. The intensity of the incident light of the system is modulated to reduce the power density on the sensor, thus avoiding sensor saturation, and improving the anti-laser interference threshold of the CASSI system. Simulation results show the improvement of reconstruction accuracy when grayscale coded apertures are employed.

Design of a Spaceborne, Compact, Off-Axis, Multi-Mirror Optical System Based on Freeform Surfaces

Based on the application requirements of high spectral resolutions, high spatial resolutions and wide swatches, a new-generation, high-performance, spaceborne, hyperspectral imaging spectrometer (NGHSI) with a spatial resolution of 15 m and a swatch of 90 km is proposed. The optical system of the NGHSI has a focal length of 1128 mm, an F-number of three, a field of view (FOV) of 7.32° and a slit length of 144 mm. A new off-axis, multi-mirror telescope structure with intermediate images is put forward, which solves the design problem that realizes secondary imaging and good telecentricity at the same time. And a new off-axis lens-compensation Offner configuration is adopted to address the challenge of the high-fidelity design of spectral imaging systems with long slit lengths. The relationship between X-Y polynomials and aberration coefficients is analyzed, and the X-Y polynomial freeform surfaces are used to correct the off-axis aberrations. The design results show that the image quality of the telescope system is close to the diffraction limit. The smile, known as the spectral distortion along the line, and keystone, which is the magnification difference for different wavelengths, of the spectral imaging system are less than 1/10 pixel size. The complete optical system of the NGHSI, including the telescope system and the spectral imaging system, has excellent imaging quality and the layout is compact and reasonable, which realizes the miniaturization design.

High-resolution calibration and data reconstruction method for image mapping spectrometer

The image mapping spectrometer (IMS) is a kind of snapshot spectral imaging system that modulates the 3D spatial-spectral datacube (x,y,λ) to a 2D sensor for simultaneous measurement. Since the IMS utilizes an image mapper for mapping the spatial information, a calibration method is required to generate the lookup table between the measured raw image pixels and the desired datacube voxels. However, the current calibration methods are either time-consuming or low-accurate. We present a combined slit scan calibration and data processing approach that can balance the calibration efficiency and accuracy, by taking in account the irregularity of field of view (FOV) of the image mapper in the calibrating setup. Additionally, we analyze the spatial-spectral mixing phenomenon of the IMS datacube recovered from the calibrated lookup table. To reconstruct high-quality spectral datacube from the spatial-spectral mixed datacube, a spatial-spectral unmixing network (SSUNet) based on multi-head multi-attention mechanism is presented. Experimental results validated the effectiveness of both the calibration strategy and the datacube restoration network.

A High Throughput Tunable Filter Module for Multispectral Imaging

AbstractTunable filters are vital not only for dynamic display but also in the miniaturized spectral imaging system. Yet, the present tunable devices possess low optical throughput, bulky volume, slow response, or complex configuration. In this paper, the study proposed and experimentally demonstrated a compact tunable filter module based on metagratings combined with liquid crystals (LC). By the synergetic effect between the birefringent LC and the dichroic metagratings, the module possesses distinct transmissive features by electrically tuning the LC material, resulting from the polarization interference effect. A simple nanoimprinting method, which is suitable for the mass production of nanostructures is adopted for low‐cost fabrication. These unique features endow the present tunable filter module with low cost, high optical throughput, and compact volume. As a proof of concept, a multispectral imager is built with this filter based on a commercial microscope. Experimental results show the system response is five times higher in optical throughput across the whole visible region than the traditional liquid crystal tunable filter and the signal‐to‐noise ratio can be improved by 7 dB. Considering the feasibility of nanoimprinting, this module paves the way for spectral imaging in low‐light conditions and on‐site scenarios such as tongue diagnosis and food quality inspection.

A lightweight and high-resolution digital integrated LVF spectral imaging system

A lightweight and high-resolution digital integrated LVF spectral imaging system

Design and Development of Large-Band Dual-MSFA Sensor Camera for Precision Agriculture.

The optimal design and construction of multispectral cameras can remarkably reduce the costs of spectral imaging systems and efficiently decrease the amount of image processing and analysis required. Also, multispectral imaging provides effective imaging information through higher-resolution images. This study aimed to develop novel, multispectral cameras based on Fabry-Pérot technology for agricultural applications such as plant/weed separation, ripeness estimation, and disease detection. Two multispectral cameras were developed, covering visible and near-infrared ranges from 380 nm to 950 nm. A monochrome image sensor with a resolution of 1600 × 1200 pixels was used, and two multispectral filter arrays were developed and mounted on the sensors. The filter pitch was 4.5 μm, and each multispectral filter array consisted of eight bands. Band selection was performed using a genetic algorithm. For VIS and NIR filters, maximum RMS values of 0.0740 and 0.0986 were obtained, respectively. The spectral response of the filters in VIS was significant; however, in NIR, the spectral response of the filters after 830 nm decreased by half. In total, these cameras provided 16 spectral images in high resolution for agricultural purposes.

ESTIMATING DRY MATTER AND TOTAL SOLUBLE CONTENT IN APPLES USING A COMMERCIAL PORTABLE HYPERSPECTRAL IMAGING SYSTEM

Abstract. The quest for rapid, non-destructive, and precise technologies for fruit quality estimation is motivated by the needs across the whole food production chain. One of the emerging technologies fulfilling these requirements is spectral imaging. However, despite documented successes, the technology is yet to become established in commercial applications. The best results reported in the literature rely on fixed, non-portable dedicated setups, and controlled light conditions, which limits the potential use cases along the food production chain. In our study, we investigate the possibility of estimating dry matter content (DMC) and total soluble content (TSC) of store-bought apples in non-regulated indoor conditions using a commercial, portable, hand-held imaging system featuring a hyperspectral camera. The acquired images are transformed into per-fruit representative spectral profiles, pre-processed, and analyzed using partial least squares (PLS), the established method in the chemometrics community. We achieved the R2 of 0.93 for TSC and 0.91 for DMC on the test dataset, with a mean absolute error of 0.71 °Brix for TSC and 0.7% for DMC, which is comparable to the state-of-the-art results presented in the literature. These results indicate that recent instrumental developments enable the deployment of spectral imaging systems in a wider range of tasks in food production, requiring portability and allowing for less stringent control of environmental conditions.

Hyperspectral imaging – a short review of methods and applications

This paper takes a look at the state-of-the-art solutions in the field of spectral imaging systems by way of application examples. It is based on a comparison of currently used systems and the challenges they face, especially in the field of high-altitude imaging and satellite imaging, are discussed. Based on our own experience, an example of hyperspectral data processing is presented. The article also discusses how modern algorithms can help in understanding the data that such images can provide.

Design and characterization of a hyperbolic-elliptical lens pair in a rapid beam steering system for single-pixel terahertz spectral imaging of the cornea.

Terahertz (THz) time-domain spectroscopy has been investigated for assessment of the hydration levels in the cornea, intraocular pressure, and changes in corneal topography. Previous efforts at THz imaging of the cornea have employed off-axis parabolic mirrors to achieve normal incidence along the spherical surface. However, this comes at the cost of an asymmetric field-of-view (FOV) and a long scan time because it requires raster-scanning of the collimated beam across the large mirror diameter. This paper proposes a solution by designing a pair of aspheric lenses that can provide a larger symmetric spherical FOV (9.6 mm) and reduce the scan time by two orders of magnitude using a novel beam-steering approach. A hyperbolic-elliptical lens was designed and optimized to achieve normal incidence and phase-front matching between the focused THz beam and the target curvature. The lenses were machined from a slab of high-density polyethylene and characterized in comparison to ray-tracing simulations by imaging several targets of similar sizes to the cornea. Our experimental results showed excellent agreement in the increased symmetric FOV and confirmed the reduction in scan time to about 3-4 seconds. In the future, this lens design process can be extended for imaging the sclera of the eye and other curved biological surfaces, such as the nose and fingers.

Dual-Channel Switchable Metasurface Filters for Compact Spectral Imaging with Deep Compressive Reconstruction.

Spectral imaging technology, which aims to capture images across multiple spectral channels and create a spectral data cube, has been widely utilized in various fields. However, conventional spectral imaging systems face challenges, such as slow acquisition speed and large size. The rapid development of optical metasurfaces, capable of manipulating light fields versatilely and miniaturizing optical components into ultrathin planar devices, offers a promising solution for compact hyperspectral imaging (HSI). This study proposes a compact snapshot compressive spectral imaging (SCSI) system by leveraging the spectral modulations of metasurfaces with dual-channel switchable metasurface filters and employing a deep-learning-based reconstruction algorithm. To achieve compactness, the proposed system integrates dual-channel switchable metasurface filters using twisted nematic liquid crystals (TNLCs) and anisotropic titanium dioxide (TiO2) nanostructures. These thin metasurface filters are closely attached to the image sensor, resulting in a compact system. The TNLCs possess a broadband linear polarization conversion ability, enabling the rapid switching of the incidence polarization state between x-polarization and y-polarization by applying different voltages. This polarization conversion facilitates the generation of two groups of transmittance spectra for wavelength-encoding, providing richer information for spectral data cube reconstruction compared to that of other snapshot compressive spectral imaging techniques. In addition, instead of employing classic iterative compressive sensing (CS) algorithms, an end-to-end residual neural network (ResNet) is utilized to reconstruct the spectral data cube. This neural network leverages the 2-frame snapshot measurements of orthogonal polarization channels. The proposed hyperspectral imaging technology demonstrates superior reconstruction quality and speed compared to those of the traditional compressive hyperspectral image recovery methods. As a result, it is expected that this technology will have substantial implications in various domains, including but not limited to object detection, face recognition, food safety, biomedical imaging, agriculture surveillance, and so on.

Predicting Quality Properties of Pears during Storage Using Hyper Spectral Imaging System

This paper presents a comprehensive analysis of the application of visible–near-infrared (Vis/NIR) spectroscopy for the estimation of various chemical attributes of pear fruit. Specifically, the paper investigates how pH, titratable acidity (TA), soluble solids content (SSC), and Vitamin C change as the pear undergoes different storage durations and temperatures. To obtain the most accurate prediction models, we applied a variety of pre-processing techniques to the acquired spectra. Notably, the combination of Savitzky-Golay (S.G.), Multiplicative Scatter Correction (MSC), and second derivatives (D2) emerged as the most effective method for predicting the fruit’s pH, with an impressive rp = 0.95 and SDR = 4.9. In contrast, combining S.G., MSC, and first derivatives (D1) yielded the most accurate predictions for TA, with a robust rp = 0.98 and SDR = 9.6. The research further delved into understanding how the storage period and temperature can significantly influence the pear fruit’s chemical properties. Our findings established that as the storage duration and temperature rise, the pH of the fruit also escalates, while TA sees a decline. The research further elucidates that prolonged storage periods and elevated temperatures lead to the pear fruit shedding its intrinsic qualities, resulting in a reduction in soluble solids and Vitamin C content. To summarize, this paper underscores the immense potential of Vis/NIR spectroscopy as a non-destructive and expedient tool for monitoring the chemical attributes of pear fruit during storage, especially when subjected to diverse temperature and time conditions. These insights not only add to the existing body of knowledge but also align with earlier research on how storage conditions can affect fruit quality.

Combining Vis-NIR and NIR Spectral Imaging Techniques with Data Fusion for Rapid and Nondestructive Multi-Quality Detection of Cherry Tomatoes.

Firmness, soluble solid content (SSC) and titratable acidity (TA) are characteristic substances for evaluating the quality of cherry tomatoes. In this paper, a hyper spectral imaging (HSI) system using visible/near-infrared (Vis-NIR) and near-infrared (NIR) was proposed to detect the key qualities of cherry tomatoes. The effects of individual spectral information and fused spectral information in the detection of different qualities were compared for firmness, SSC and TA of cherry tomatoes. Data layer fusion combined with multiple machine learning methods including principal component regression (PCR), partial least squares regression (PLSR), support vector regression (SVR) and back propagation neural network (BP) is used for model training. The results show that for firmness, SSC and TA, the determination coefficient R2 of the multi-quality prediction model established by Vis-NIR spectra is higher than that of NIR spectra. The R2 of the best model obtained by SSC and TA fusion band is greater than 0.9, and that of the best model obtained by the firmness fusion band is greater than 0.85. It is better to use the spectral bands after information fusion for nondestructive quality detection of cherry tomatoes. This study shows that hyperspectral imaging technology can be used for the nondestructive detection of multiple qualities of cherry tomatoes, and the method based on the fusion of two spectra has a better prediction effect for the rapid detection of multiple qualities of cherry tomatoes compared with a single spectrum. This study can provide certain technical support for the rapid nondestructive detection of multiple qualities in other melons and fruits.

Compressive hyperspectral microscopy for cancer detection.

Hyperspectral microscopy grants the ability to characterize unique properties of tissues based on their spectral fingerprint. The ability to label and measure multiple molecular probes simultaneously provides pathologists and oncologists with a powerful tool to enhance accurate diagnostic and prognostic decisions. As the pathological workload grows, having an objective tool that provides companion diagnostics is of immense importance. Therefore, fast whole-slide spectral imaging systems are of immense importance for automated cancer prognostics that meet current and future needs. We aim to develop a fast and accurate hyperspectral microscopy system that can be easily integrated with existing microscopes and provide flexibility for optimizing measurement time versus spectral resolution. The method employs compressive sensing (CS) and a spectrally encoded illumination device integrated into the illumination path of a standard microscope. The spectral encoding is obtained using a compact liquid crystal cell that is operated in a fast mode. It provides time-efficient measurements of the spectral information, is modular and versatile, and can also be used for other applications that require rapid acquisition of hyperspectral images. We demonstrated the acquisition of breast cancer biopsies hyperspectral data of the whole camera area within . This means that a typical biopsy can be measured in . The hyperspectral images with 250 spectral bands are reconstructed from 47 spectrally encoded images in the spectral range of 450 to 700nm. CS hyperspectral microscopy was successfully demonstrated on a common lab microscope for measuring biopsies stained with the most common stains, such as hematoxylin and eosin. The high spectral resolution demonstrated here in a rather short time indicates the ability to use it further for coping with the highly demanding needs of pathological diagnostics, both for cancer diagnostics and prognostics.

Real-Time Segmentation of Artificial Targets Using a Dual-Modal Efficient Attention Fusion Network

The fusion of spectral–polarimetric information can improve the autonomous reconnaissance capability of unmanned aerial vehicles (UAVs) in detecting artificial targets. However, the current spectral and polarization imaging systems typically suffer from low image sampling resolution, which can lead to the loss of target information. Most existing segmentation algorithms neglect the similarities and differences between multimodal features, resulting in reduced accuracy and robustness of the algorithms. To address these challenges, a real-time spectral–polarimetric segmentation algorithm for artificial targets based on an efficient attention fusion network, called ESPFNet (efficient spectral–polarimetric fusion network) is proposed. The network employs a coordination attention bimodal fusion (CABF) module and a complex atrous spatial pyramid pooling (CASPP) module to fuse and enhance low-level and high-level features at different scales from the spectral feature images and the polarization encoded images, effectively achieving the segmentation of artificial targets. Additionally, the introduction of the residual dense block (RDB) module refines feature extraction, further enhancing the network’s ability to classify pixels. In order to test the algorithm’s performance, a spectral–polarimetric image dataset of artificial targets, named SPIAO (spectral–polarimetric image of artificial objects) is constructed, which contains various camouflaged nets and camouflaged plates with different properties. The experimental results on the SPIAO dataset demonstrate that the proposed method accurately detects the artificial targets, achieving a mean intersection-over-union (MIoU) of 80.4%, a mean pixel accuracy (MPA) of 88.1%, and a detection rate of 27.5 frames per second, meeting the real-time requirement. The research has the potential to provide a new multimodal detection technique for enabling autonomous reconnaissance by UAVs in complex scenes.

Prior Image Guided Snapshot Compressive Spectral Imaging.

Spectral images with rich spatial and spectral information have wide usage, however, traditional spectral imaging techniques undeniably take a long time to capture scenes. We consider the computational imaging problem of the snapshot spectral spectrometer, i.e., the Coded Aperture Snapshot Spectral Imaging (CASSI) system. For the sake of a fast and generalized reconstruction algorithm, we propose a prior image guidance-based snapshot compressive imaging method. Typically, the prior image denotes the RGB measurement captured by the additional uncoded panchromatic camera of the dual-camera CASSI system. We argue that the RGB image as a prior image can provide valuable semantic information. More importantly, we design the Prior Image Semantic Similarity (PIDS) regularization term to enhance the reconstructed spectral image fidelity. In particular, the PIDS is formulated as the difference between the total variation of the prior image and the recovered spectral image. Then, we solve the PIDS regularized reconstruction problem by the Alternating Direction Method of Multipliers (ADMM) optimization algorithm. Comprehensive experiments on various datasets demonstrate the superior performance of our method.

Multi-Parameter Model-Based Polarimetric Calibration for Dual-Coded Spectral Polarization Imaging System

A polarization analysis method based on a multi-parameter model is proposed to address the polarization effect analysis and calibration requirements of a dual-coded snapshot spectral polarization imaging system. A full-link polarization effect model for a spectral polarization imaging system is established that includes a digital micromirror array (DMD), prism grating prism (PGP), micro-polarizer array detector (MPA), and multi-film. The influence of parameters such as the refractive index, incident angle, grating refractive index, constant, prism refractive index, vertex angle, multi-layer film complex refractive index, and film thickness on the optical transmittance of the system are analyzed. Using a dynamic data exchange mechanism to perform full-link, full-FOV, and full-pupil ray tracing on the optical system, the polarization effect distribution of the system under different degrees of polarization (DOP) and wavelengths is obtained. A calibration experiment for the controllable incident wavelength and DOP using narrowband filters and glass stacks is established. The experimental results show that in the 420 nm, 532 nm, and 635 nm wavelength bands, the MSEs of the calibrated values are 1.3924 × 10−4, 1.6852 × 10−4, and 1.6735 × 10−4, respectively. It is proven that the calibration method based on a multi-parameter model is feasible. Finally, the spectral polarization image at 532 nm is calibrated. The contrast ratio of metallic aluminum is calibrated from 7.13 to 15.33. This study provides a theoretical basis for the analysis and calibration of polarization effects in a dual-coded snapshot spectral polarization imaging system.

Interferometer response characterization algorithm for multi-aperture Fabry-Perot imaging spectrometers.

In recent years, the demand for hyperspectral imaging devices has grown significantly, driven by their ability of capturing high-resolution spectral information. Among the several possible optical designs for acquiring hyperspectral images, there is a growing interest in interferometric spectral imaging systems based on division of aperture. These systems have the advantage of capturing snapshot acquisitions while maintaining a compact design. However, they require a careful calibration to operate properly. In this work, we present the interferometer response characterization algorithm (IRCA), a robust three-step procedure designed to characterize the transmittance response of multi-aperture imaging spectrometers based on the interferometry of Fabry-Perot. Additionally, we propose a formulation of the image formation model for such devices suitable to estimate the parameters of interest by considering the model under various regimes of finesse. The proposed algorithm processes the image output obtained from a set of monochromatic light sources and refines the results using nonlinear regression after an ad-hoc initialization. Through experimental analysis conducted on four different prototypes from the Image SPectrometer On Chip (ImSPOC) family, we validate the performance of our approach for characterization. The associated source code for this paper is available from Zenodo (http://dx.doi.org/10.5281/zenodo.7978514).