Hyperspectral Reconstruction Research Articles

Robust hyperspectral reconstruction via a multi-channel clustering compressive sensing approach

Wavelength-coded spectral imaging represents a fusion of spectral imaging and compressed sensing, offering advantages such as reduced storage requirements and straightforward miniaturization. This approach employs optical filters for hyperspectral reconstruction. However, issues like insufficient energy and excessive noise arise in low-light detection, leading to a notable degradation of image reconstruction quality. This paper presents a robust segmented reconstruction algorithm named MCC-WSCI (Multi-Channel Clustering-based Wavelength-Coded Spectral Imaging), designed to enhance the method's tolerance to noise. This algorithm can accurately classify and process compressed images obtained by wavelength-coded spectral imaging systems, thus improving reconstruction quality significantly. Numerical simulations and experiments in low-light scenarios are carried out to verify the proposed method. Results show that the MCC-WSCI method is robust to different noise and sampling rates and outperforms other state-of-the-art compressed sensing reconstruction methods in terms of reconstructed spatial resolution and spectral resolution. The proposed method provides effective experimental robustness to wavelength-coded spectral imaging with a natural algorithmic extension, paving the way for its application in remote sensing.

Comparative analysis of hyperspectral Image reconstruction using deep learning for agricultural and biological applications

Hyperspectral imaging (HSI) has become a key technology for non-invasive quality evaluation in various fields, offering detailed insights through spatial and spectral data. Despite its efficacy, the complexity and high cost of HSI systems have hindered their widespread adoption. This study addressed these challenges by exploring deep learning-based hyperspectral image reconstruction from RGB (Red, Green, Blue) images, particularly for agricultural products. Specifically, different hyperspectral reconstruction algorithms, such as Hyperspectral Convolutional Neural Network - Dense (HSCNN-D), High-Resolution Network (HRNET), and Multi-Scale Transformer Plus Plus (MST++), were compared to assess the dry matter content of sweet potatoes. Among the tested reconstruction methods, HRNET demonstrated superior performance, achieving the lowest mean relative absolute error (MRAE) of 0.07, root mean square error (RMSE) of 0.03, and the highest peak signal-to-noise ratio (PSNR) of 32.28 decibels (dB). Some key features were selected using the genetic algorithm (GA), and their importance was interpreted using explainable artificial intelligence (XAI). Partial least squares regression (PLSR) models were developed using the RGB, reconstructed, and ground truth (GT) data. The visual and spectra quality of these reconstructed methods was compared with GT data, and predicted maps were generated. The results revealed the prospect of deep learning-based hyperspectral image reconstruction as a cost-effective and efficient quality assessment tool for agricultural and biological applications.

Acne Detection Based on Reconstructed Hyperspectral Images.

Acne Vulgaris is a common type of skin disease that affects more than 85% of teenagers and frequently continues even in adulthood. While it is not a dangerous skin disease, it can significantly impact the quality of life. Hyperspectral imaging (HSI), which captures a wide spectrum of light, has emerged as a tool for the detection and diagnosis of various skin conditions. However, due to the high cost of specialised HS cameras, it is limited in its use in clinical settings. In this research, a novel acne detection system that will utilise reconstructed hyperspectral (HS) images from RGB images is proposed. A dataset of reconstructed HS images is created using the best-performing HS reconstruction model from our previous research. A new acne detection algorithm that is based on reconstructed HS images and RetinaNet algorithm is introduced. The results indicate that the proposed algorithm surpasses other techniques based on RGB images. Additionally, reconstructed HS images offer a promising and cost-effective alternative to using expensive HSI equipment for detecting conditions like acne or other medical issues.

Attention and transformer complementary fusion network for hyperspectral image spectral reconstruction

ABSTRACT Hyperspectral images can be widely used in many fields due to their high information richness. However, costly and complex imaging spectrometers limit its growth. Hyperspectral reconstruction aims to obtain the corresponding hyperspectral image from the multispectral image, to reduce the acquisition cost of hyperspectral images. At present, the related works mainly use methods based on deep learning, and they have achieved good results. However, how to fully extract the global spectral and spatial features of hyperspectral images is still not well solved. To address these issues, we propose an Attention and Transformer Complementary Fusion Network (ATCFNet), which is composed of three modules: Multi-angle Input Image Processing (MIIP), Deep Feature Extraction (DFE) and Hyperspectral Reconstruction (HR) modules. Within the DFE module, an improved Transformer module and a novel Multi-scale Spatial Attention (MSA) module are proposed to extract the global spectral relationship and the spatial features of the hyperspectral images, respectively. Moreover, the MIIP module is proposed to extract the features of input multispectral images more effectively and comprehensively. To verify these, we compare the proposed ATCFNet with other excellent reconstruction methods on three hyperspectral image datasets. The results show that our method achieves the best results in these datasets. Code is publicly available at https://github.com/kidder314/ATCFNet.

Hyperspectral Image Reconstruction via Combinatorial Embedding of Cross-Channel Spatio-Spectral Clues

Existing learning-based hyperspectral reconstruction methods show limitations in fully exploiting the information among the hyperspectral bands. As such, we propose to investigate the chromatic inter-dependencies in their respective hyperspectral embedding space. These embedded features can be fully exploited by querying the inter-channel correlations in a combinatorial manner, with the unique and complementary information efficiently fused into the final prediction. We found such independent modeling and combinatorial excavation mechanisms are extremely beneficial to uncover marginal spectral features, especially in the long wavelength bands. In addition, we have proposed a spatio-spectral attention block and a spectrum-fusion attention module, which greatly facilitates the excavation and fusion of information at both semantically long-range levels and fine-grained pixel levels across all dimensions. Extensive quantitative and qualitative experiments show that our method (dubbed CESST) achieves SOTA performance. Code for this project is at: https://github.com/AlexYangxx/CESST.

Learning a physics-based filter attachment for hyperspectral imaging with RGB cameras

Countless RGB cameras are ubiquitously distributed in our daily lives, serving to perceive and depict the diverse colors of the world. Reconstructing hyperspectral images (HSI) from these trichromatic cameras emerges as a promising solution to address the limitations of existing, costly hyperspectral imaging systems. The performance of HSI reconstruction relies heavily on the camera spectral response (CSR). Thus, designing a better CSR and putting it into practice is the critical issue for RGB-based HSI reconstruction. However, the CSR curves designed in the existing works are overly random, making them challenging to manufacture directly. Additionally, the designed CSR curves require modifications to the camera hardware, resulting in the loss of RGB imaging functionality. In this paper, we propose a hyperspectral imaging system, which involves enhancing the CSR curve of existing RGB cameras and preserving RGB imaging functionality by adding a learnable physics-based spectral filter. Specifically, we first parameterize the spectral filter transmittance as a function of the filter thicknesses, based on the physical constraints of the multilayer interference principle. Then, we propose a joint optimization framework in which the thicknesses of the filter and the hyperspectral reconstruction network are optimized. In this manner, the thicknesses of the filter are obtained and used to manufacture the filter directly. Finally, we construct a prototype and verify the benefits of our spectral filter design method through experiments including both synthetic data and real images.

Investigating the impact of hyperspectral reconstruction techniques on the quantitative inversion of rice physiological parameters: A case study using the MST++ model

Quantitative inversion is a significant topic in remote sensing science. The development of visible light-based hyperspectral reconstruction techniques has opened up novel prospects for low-cost, high-precision remote sensing inversion in agriculture. The aim of this study was to assesses the effectiveness of hyperspectral reconstruction technology in agricultural remote sensing applications. Hyperspectral images were reconstructed using the MST++ hyperspectral reconstruction model and compared with the original visible light images in terms of their correlation with physiological parameters, the accuracy of single-feature modeling, and the accuracy of combined feature modeling. The results showed that compared to the visible light image, the reconstructed data exhibited a stronger correlation with physiological parameters, and the accuracy was improved in both the single-feature and the combined feature inversion mode. However, compared to multispectral sensors, hyperspectral reconstruction provided limited improvement on the inversion model accuracy. It was concluded that for physiological parameters that are not easy to be directly observed, deep mining of features in visible light. data through hyperspectral reconstruction technology can improve the accuracy of the inversion model. Appropriate feature selection and simple models are more suitable for the remote sensing inversion task of traditional agronomic plot experiments. To strengthen the application of hyperspectral reconstruction technology in agricultural remote sensing, further development is necessary with broader wavelength ranges and more diverse agricultural scenes.

Bi-directional hyperspectral reconstruction of cherry tomato: diagnosis of internal tissues maturation stage and composition.

Precision monitoring maturity in climacteric fruits like tomato is crucial for minimising losses within the food supply chain and enhancing pre- and post-harvest production and utilisation. This paper introduces an approach to analyse the precision maturation of tomato using hyperspectral tomography-like. A novel bi-directional spectral reconstruction method is presented, leveraging visible to near-infrared (Vis-NIR) information gathered from tomato spectra and their internal tissues (skin, pulp, and seeds). The study, encompassing 118 tomatoes at various maturation stages, employs a multi-block hierarchical principal component analysis combined with partial least squares for bi-directional reconstruction. The approach involves predicting internal tissue spectra by decomposing the overall tomato spectral information, creating a superset with eight latent variables for each tissue. The reverse process also utilises eight latent variables for reconstructing skin, pulp, and seed spectral data. The reconstruction of the tomato spectra presents a mean absolute percentage error of 30.44 % and 5.37 %, 5.25 % and 6.42 % and Pearson's correlation coefficient of 0.85, 0.98, 0.99 and 0.99 for the skin, pulp and seed, respectively. Quality parameters, including soluble solid content (%), chlorophyll (a.u.), lycopene (a.u.), and puncture force (N), were assessed and modelled with PLS with the original and reconstructed datasets, presenting a range of R2 higher than 0.84 in the reconstructed dataset. An empirical demonstration of the tomato maturation in the internal tissues revealed the dynamic of the chlorophyll and lycopene in the different tissues during the maturation process. The proposed approach for inner tomato tissue spectral inference is highly reliable, provides early indications and is easy to operate. This study highlights the potential of Vis-NIR devices in precision fruit maturation assessment, surpassing conventional labour-intensive techniques in cost-effectiveness and efficiency. The implications of this advancement extend to various agronomic and food chain applications, promising substantial improvements in monitoring and enhancing fruit quality.

High-precision hemoglobin detection based on hyperspectral reconstruction of RGB images

Hemoglobin concentration is an important indicator for evaluating human health. The concentration value is an important evidence for early diagnosis of anemia and other blood diseases. However, existing optical detection methods usually suffer from low precision, strong noise and tedious operation, makes it difficult to obtain an accurate hemoglobin concentration evaluation. In this work, we propose and demonstrate a method for hemoglobin detection with the hyperspectral reconstruction of RGB images. To be specific, this overall model consists of a SHR-Net (Skin Hyperspectral Reflectance Network) to reconstruct hyperspectral images from RGB images of fingers, a feature selection process to capture full-band reflectance spectra based on the maximum blood perfusion region and an SVR (Support Vector Regression) algorithm to predict the hemoglobin concentration. The experimental results show that the coefficient of determination between the predicted and true values is 0.945. The mean square error of cross-validation is 0.49 g/dL. Therefore, this method provides a new solution for the accurate and noninvasive measurement of hemoglobin.

HSGAN: Hyperspectral Reconstruction From RGB Images With Generative Adversarial Network.

Hyperspectral (HS) reconstruction from RGB images denotes the recovery of whole-scene HS information, which has attracted much attention recently. State-of-the-art approaches often adopt convolutional neural networks to learn the mapping for HS reconstruction from RGB images. However, they often do not achieve high HS reconstruction performance across different scenes consistently. In addition, their performance in recovering HS images from clean and real-world noisy RGB images is not consistent. To improve the HS reconstruction accuracy and robustness across different scenes and from different input images, we present an effective HSGAN framework with a two-stage adversarial training strategy. The generator is a four-level top-down architecture that extracts and combines features on multiple scales. To generalize well to real-world noisy images, we further propose a spatial-spectral attention block (SSAB) to learn both spatial-wise and channel-wise relations. We conduct the HS reconstruction experiments from both clean and real-world noisy RGB images on five well-known HS datasets. The results demonstrate that HSGAN achieves superior performance to existing methods. Please visit https://github.com/zhaoyuzhi/HSGAN to try our codes.

Hyperspectral Reconstruction from RGB Images via Physical Guided Graph Deep Prior Learning

Hyperspectral Reconstruction from RGB Images via Physical Guided Graph Deep Prior Learning

An improved sea surface salinity retrieval algorithm for the Chinese Bohai Sea based on hyperspectral reconstruction and its applicability analysis

Salinity is regarded as a critical factor in marine systems. It is essential to monitor the long-term changes in sea surface salinity (SSS), as well as its short-term changes. Along these lines, in this work, to monitor the hourly, daily, and monthly changes in SSS in the Chinese Bohai Sea from GOCI (Geostationary Ocean Color Imager) images, the feasibility of adopting a MERIS-based SSS retrieval algorithm was systematically explored to develop a GOCI-based SSS retrieval algorithm. To avoid the influence of the atmospheric correction error on Rrs(λ) of the similar bands of MERIS (490 nm, 560 nm, and 665 nm) and those of GOCI (490 nm, 555 nm, and 660 nm), a simple linear regression model was applied based on the ρrs(λ) matchup of the similar bands by using a hyperspectral reconstruction method with the measured ρ(λ). Despite the aerosol-based scattering errors, a more reasonable result (R2 = 0.86, RMSE = 0.79 psu) over a long time scale was gained by the improved SSS retrieval algorithm, which was validated by the measured data (R2 = 0.92, RMSE = 0.72 psu). On top of that, the temporal and spatial distribution of SSS, as well as its diurnal, daily, and monthly changes, were then obtained in the Chinese Bohai Sea from April 2011 to March 2012. The impact of runoff on SSS was also analysed based on instant and lag correlations.

Combination of super-resolution reconstruction and SGA-Net for marsh vegetation mapping using multi-resolution multispectral and hyperspectral images

ABSTRACT Vegetation is crucial for wetland ecosystems. Human activities and climate changes are increasingly threatening wetland ecosystems. Combining satellite images and deep learning for classifying marsh vegetation communities has faced great challenges because of its coarse spatial resolution and limited spectral bands. This study aimed to propose a method to classify marsh vegetation using multi-resolution multispectral and hyperspectral images, combining super-resolution techniques and a novel self-constructing graph attention neural network (SGA-Net) algorithm. The SGA-Net algorithm includes a decoding layer (SCE-Net) to precisely fine marsh vegetation classification in Honghe National Nature Reserve, Northeast China. The results indicated that the hyperspectral reconstruction images based on the super-resolution convolutional neural network (SRCNN) obtained higher accuracy with a peak signal-to-noise ratio (PSNR) of 28.87 and structural similarity (SSIM) of 0.76 in spatial quality and root mean squared error (RMSE) of 0.11 and R2 of 0.63 in spectral quality. The improvement of classification accuracy (MIoU) by enhanced super-resolution generative adversarial network (ESRGAN) (6.19%) was greater than that of SRCNN (4.33%) and super-resolution generative adversarial network (SRGAN) (3.64%). In most classification schemes, the SGA-Net outperformed DeepLabV3 + and SegFormer algorithms for marsh vegetation and achieved the highest F1-score (78.47%). This study demonstrated that collaborative use of super-resolution reconstruction and deep learning is an effective approach for marsh vegetation mapping.

Single-Slice XRF Mapping of Light Elements in Frozen-Hydrated Allium schoenoprasum via a Self-Absorption-Corrected Hyperspectral Tomographic Reconstruction Approach.

3D and 2D-cross-sectional X-ray fluorescence analysis of biological material is a powerful tool to image the distribution of elements and to understand and quantify metal homeostasis and the distribution of anthropogenic metals and nanoparticles with minimal preparation artifacts. Using tomograms recorded on cryogenically prepared leaves of Allium schoenoprasum, the cross-sectional distribution of physiologically relevant elements like calcium, potassium, manganese, and zinc could be tomographically reconstructed by peak fitting followed by a conventional maximum-likelihood algorithm with self-absorption correction to reveal the quantitative cross-sectional element distribution. If light elements such as S and P are located deep in the sample compared to the escape depth of their characteristic X-ray fluorescence lines, the quantitative reconstruction becomes inaccurate. As a consequence, noise is amplified to a magnitude where it might be misinterpreted as actual concentration. We show that a tomographic MCA hyperspectral reconstruction in combination with a self-absorption correction allows for fitting of the XRF spectra directly in real space, which significantly improves the qualitative and quantitative analysis of the light elements compared to the conventional method as noise and artifacts in the tomographic reconstruction are reduced. This reconstruction approach can substantially improve the quantitative analysis of trace elements as it allows the fitting of summed voxel spectra in anatomical regions of interest. The presented method can be applied to XRF 2D single-slice tomography data and 3D tomograms and is particularly relevant for, but not limited to, biological material in order to help retrieve self-absorption corrected quantitative reconstructions of the spatial distribution of light elements and ultra-trace-elements.

Snapshot hyperspectral imaging based on equalization designed DOE.

Hyperspectral imaging attempts to determine distinctive information in spatial and spectral domain of a target. Over the past few years, hyperspectral imaging systems have developed towards lighter and faster. In phase-coded hyperspectral imaging systems, a better coding aperture design can improve the spectral accuracy relatively. Using wave optics, we post an equalization designed phase-coded aperture to achieve desired equalization point spread functions (PSFs) which provides richer features for subsequent image reconstruction. During the reconstruction of images, our raised hyperspectral reconstruction network, CAFormer, achieves better results than the state-of-the-art networks with less computation by substituting self-attention with channel-attention. Our work revolves around the equalization design of the phase-coded aperture and optimizes the imaging process from three aspects: hardware design, reconstruction algorithm, and PSF calibration. Our work is putting snapshot compact hyperspectral technology closer to a practical application.

Dual-camera compressive hyperspectral imaging based on deep image prior and a guided filter.

Coded aperture snapshot spectral imaging (CASSI) aims to capture the high-dimensional (usually 3D) data cube using a 2D sensor in a single snapshot. Due to the ill-posed snapshot, the reconstruction results are not ideal. One feasible solution is to utilize additional information such as the panchromatic measurement in CASSI. In this paper, we propose a dual-camera hyperspectral reconstruction method based on the deep image prior (DIP) and a guided filter. In particular, the panchromatic measurements are used to estimate spatial detail, and spectral details are provided using CASSI measurements. These measurements are used as a priori learning by the self-supervised network. Using iteration combined with DIP, the hyperspectral reconstruction is continuously updated iteratively. Finally, the panchromatic measurement is used as the guidance image, and the reconstruction result is optimized by guide filtering. A large number of experimental results demonstrate that our method without training data can reconstruct spectral data with both high spectral accuracy and spatial resolution.

Actively Tunable Spectral Filter for Compact Hyperspectral Camera Using Angle‐Sensitive Plasmonic Structures

AbstractHyperspectral imaging provides enhanced classification and identification of veiled features for diverse biomedical applications such as label‐free cancer detection or non‐invasive vascular disease diagnostics. However, hyperspectral cameras still have technical limitations in miniaturization due to the inherently complex and bulky configurations of conventional tunable filters. Herein, a compact hyperspectral camera using an active plasmonic tunable filter (APTF) with electrothermally driven spectral modulation for feature‐augmented imaging is reported. APTF consists of angle‐sensitive plasmonic structures (APS) over an electrothermal MEMS (Microelectromechanical systems) actuator, fabricated by combining nanoimprint lithography, and MEMS fabrication ona 6‐inch wafer. APS have a complementary configuration of Au nanohole and nanodisk arrays supported on asilicon nitride membrane. APTF shows a large angular motion at operational voltages of 5–9 VDC for continuous spectral modulation between 820 and 1000 nm (45 nm/V). The compact hyperspectral camera was fully packaged with a linear polarizer, APTF and amonochromatic camera, exhibiting asize of 16 mm (ϕ) × 9.5 mm (h). Feature‐augmented images of subcutaneous vein and a fresh fruit have been successfully demonstrated after the hyperspectral reconstruction and spectral feature extraction. This functional camera provides a new compact platform for point‐of‐care or in vivo hyperspectral imaging in biomedical applications.

Hyperspectral image reconstruction from colored natural flame luminosity imaging in a tri-fuel optical engine

The detection of chemiluminescence from various radicals and molecules in a hydrocarbon flame can provide valuable information on the rate of local heat release, combustion stability, and combustion completeness. In this study, chemiluminescence from the combustion process is detected using a high-speed color camera within the broadband spectrum of visible light. Whereon, a novel hyperspectral reconstruction approach based on the physically plausible spectral reconstruction (PPSR) is employed to reconstruct the spectral chemiluminescence signals from 400 to 700 nm with a resolution of 10 nm to provide 31 different spectral channels. The reconstructed key chemiluminescence signals (e.g., CH*, CH2O*, C2*, and CO2*) from the color images are further analyzed to characterize the chemical kinetics and combustion processes under engine conditions. The spectral chemiluminescence evolution with engine crank angle is identified to comprehend the effect of H2 fraction on flame characteristics and combustion kinetics. Additionally, in this study, a detailed kinetic mechanism is adopted to deepen the theoretical understanding and describe the spectral chemiluminescence from H2/CH4 and H2/CH4/n-dodecane flames at relevant conditions for various species including OH*, CH*, C2*, and CO2*. The results indicate that the PPSR is an adequately reliable approach to reconstructing spectral wavelengths based on chemiluminescence signals from the color images, which can potentially provide qualitative information about the evolution of various species during combustion. Here, the reconstructed chemiluminescence images show less than 1% errors compared to the raw images in red, green, and blue channels. Furthermore, the reconstructed chemiluminescence trends of CH*, CH2O*, C2*, and CO2* show a good agreement with the detailed kinetics 0D simulation.

A hyperspectral image reconstruction algorithm based on RGB image using multi-scale atrous residual convolution network

Hyperspectral images are a valuable tool for remotely sensing important characteristics of a variety of landscapes, including water quality and the status of marine disasters. However, hyperspectral data are rare or expensive to obtain, which has spurred interest in low-cost, fast methods for reconstructing hyperspectral data from much more common RGB images. We designed a novel algorithm to achieve this goal using multi-scale atrous convolution residual network (MACRN). The algorithm includes three parts: low-level feature extraction, high-level feature extraction, and feature transformation. The high-level feature extraction module is composed of cascading multi-scale atrous convolution residual blocks (ACRB). It stacks multiple modules to form a depth network for extracting high-level features from the RGB image used as an input. The algorithm uses jump connection for residual learning, and the final high-level feature combines the output of the low-level feature extraction module and the output of the cascaded atrous convolution residual block element by element, so as to prevent gradient dispersion and gradient explosion in the deep network. Without adding too many parameters, the model can extract multi-scale features under different receptive fields, make better use of the spatial information in RGB images, and enrich the contextual information. As a proof of concept, we ran an experiment using the algorithm to reconstruct hyperspectral Sentinel-2 satellite data from the northern coast of Australia. The algorithm achieves hyperspectral spectral reconstruction in 443nm-2190nm band with less computational cost, and the results are stable. On the Realworld dataset, the reconstruction error MARE index is less than 0.0645, and the reconstruction time is less than 9.24S. Therefore, in the near infrared band, MACRN reconstruction accuracy is significantly better than other spectral reconstruction algorithms. MACRN hyperspectral reconstruction algorithm has the characteristics of low reconstruction cost and high reconstruction accuracy, and its advantages in ocean spectral reconstruction are more obvious.

End-to-end joint optimization of metasurface and image processing for compact snapshot hyperspectral imaging

Traditional snapshot hyperspectral imaging systems generally require multiple refractive-optics-based elements to modulate light, resulting in bulky framework. In pursuit of a more compact form factor, a metasurface-based snapshot hyperspectral imaging system, which achieves joint optimization of metasurface and image processing, is proposed in this paper. The unprecedented light manipulation capabilities of metasurfaces are used in conjunction with neural networks to encode and decode light fields for better hyperspectral imaging. Specifically, the extremely strong dispersion of metasurfaces is exploited to distinguish spectral information, and a neural network based on spectral priors is applied for hyperspectral image reconstruction. By constructing a fully differentiable model of metasurface-based hyperspectral imaging, the front-end metasurface phase distribution and the back-end recovery network parameters can be jointly optimized. This method achieves high-quality hyperspectral reconstruction results numerically, outperforming separation optimization methods. The proposed system holds great potential for miniaturization and portability of hyperspectral imaging systems.