Hyperspectral Image Reconstruction Research Articles

FSL-Unet: Full-Scale Linked Unet With Spatial–Spectral Joint Perceptual Attention for Hyperspectral and Multispectral Image Fusion

The application of hyperspectral image (HSI) is more and more extensive, but the lower spatial resolution seriously affects its application effect. Using low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (MSI) fusion technology to achieve super-resolution reconstruction of HSI has become a mainstream method. However, most of the existing fusion methods do not make full use of the large-scale range of remote sensing images, and neglect the preservation of spatial-spectral information in the fusion process. Considering that the spectral information in fused high-resolution hyperspectral image (HR-HSI) mainly depends on HSI, and the spatial information mainly depends on MSI, this paper proposes a full-scale linked Unet with spatial-spectral joint perceptual attention for hyperspectral and multispectral image fusion (FSL-Unet). The FSL-Unet consists of two modules, the first is spatial-spectral attention extraction module (SSAE), which is used to calculate the spectral attention of LR-HSI and the spatial attention of HR-MSI at different scales. The second is the full-scale link U-shaped fusion module (FLUF), which adopts a multi-level feature extraction strategy, using denser full-scale skip connections to explore feature information in a finer-grained range, enabling flexible combination of multi-scale and multi-path features. At the same time, we propose spatial-spectral joint peceptual attention (SSJPA) on the encoder side of FLUF. SSJPA can make full use of the attention maps computed by the SSAE, and then effectively embed spatial and spectral information into the fused image, enabling uninterrupted information transfer and aggregation. To demonstrate the effectiveness of FSL-Unet, we selected five public hyperspectral datasets for experiments. Compared with other eight state-of-the-art fusion methods, the experimental results show that the FSL-Unet achieves competitive results. The source code for FSL-Unet can be downloaded from https://github.com/wxy11-27/FSL-Unet.

High-Accuracy Image Formation Model for Coded Aperture Snapshot Spectral Imaging

Coded aperture snapshot spectral imaging (CASSI) is based on the binary modulation of the spatial-spectral scene, which allows for hyperspectral image reconstruction from 2D compressive measurement. However, the actual optical modulation does not match the current image formation model due to the extra optical phenomena, such as diffraction, distortion, optical misalignment, and dispersion, inside the system. It is a long-lasting problem that the gap between the simplified image formation model and the actual optical modulation degrades the reconstruction quality. In this paper, we propose a high-accuracy image formation model to reduce this gap in CASSI. Specifically, we first reformulate the spectral modulation as channel-wise convolution, in which the convolution kernel represents the point-spread-function (PSF) of each spectral channel. Then, according to our key observation that the calibration images are the blurred versions of the coded aperture, we propose to estimate the PSF by exploring the relationship between these blurred and non-blurred pairs. In addition, we also provide a theoretical analysis of the PSF’s influences on the reconstruction quality, which can serve as a guide for CASSI system implementation. Our simulations and real system experiments demonstrate the effectiveness of the proposed model.

Super-Resolution Hyperspectral Reconstruction With Majorization-Minimization Algorithm and Low-Rank Approximation

Hyperspectral imaging (HSI) has become an invaluable imaging tool for many applications in astrophysics or Earth observation. Unfortunately, direct observation of hyperspectral images is impossible since the actual measurements are 2-D and suffer from strong spatial and spectral degradations, especially in the infrared. We present in this work an original method for high-resolution hyperspectral image reconstruction from heterogeneous 2-D measurements degraded by integral field spectroscopy (IFS) instrument. A fundamental part of this work is developing a forward model that accounts for the limitations of the IFS instrument, such as wavelength-dependent spatial and spectral blur, subsampling, and inhomogeneous sampling steps. The reconstruction method inverts the forward model using a deterministic regularization framework for edge-preserving. It fuses information from different observations and spectral bands for resolution enhancements. We rely on the Majorize-Minimize memory gradient (3MG) optimization algorithm to solve the inverse problem while considering a low-rank approximation for the unknown to handle the high-dimensionality of the problem.

An Efficient Cross-Modality Self-Calibrated Network for Hyperspectral and Multispectral Image Fusion

Recently, deep convolutional neural network based hyperspectral and multispectral image fusion methods have shown significant performance. Nevertheless, the rich spatial and spectral details of hyperspectral images (HSIs) have not been fully explored, leaving room for further improve the representation ability of the model. In this paper, we propose an efficient cross-modality self-calibrated network (CMSCN) for hyperspectral and multispectral image fusion. Specifically, we use a cross-modality non-local module to fuse a high-resolution multispectral image (HR-MSI) and a low-resolution hyperspectral image (LR-HSI) to get an enhanced LR-HSI. In addition, a novel cross-scale self-calibrated convolution structure is proposed to explore and exploit multi-scale and hierarchical spatial-spectral features, which can improve the learning ability of the model. The introduced efficient spatial-spectral attention mechanism can calibrate the feature representation at different dimensions, thereby providing more efficient and accurate information for hyperspectral image reconstruction. Extensive experimental results on various hyperspectral images demonstrate the superiority of our method in comparison with the state-of-the-art image fusion methods.

Hyperspectral Image Reconstruction Based on the Fusion of Diffracted Rotation Blurred and Clear Images

Hyperspectral Image Reconstruction Based on the Fusion of Diffracted Rotation Blurred and Clear Images

3D denoised completion network for deep single-pixel reconstruction of hyperspectral images.

Single-pixel imaging acquires an image by measuring its coefficients in a transform domain, thanks to a spatial light modulator. However, as measurements are sequential, only a few coefficients can be measured in the real-time applications. Therefore, single-pixel reconstruction is usually an underdetermined inverse problem that requires regularization to obtain an appropriate solution. Combined with a spectral detector, the concept of single-pixel imaging allows for hyperspectral imaging. While each channel can be reconstructed independently, we propose to exploit the spectral redundancy between channels to regularize the reconstruction problem. In particular, we introduce a denoised completion network that includes 3D convolution filters. Contrary to black-box approaches, our network combines the classical Tikhonov theory with the deep learning methodology, leading to an explainable network. Considering both simulated and experimental data, we demonstrate that the proposed approach yields hyperspectral images with higher quantitative metrics than the approaches developed for grayscale images.

Coded Aperture Hyperspectral Image Reconstruction.

In this work, we study and analyze the reconstruction of hyperspectral images that are sampled with a CASSI device. The sensing procedure was modeled with the help of the CS theory, which enabled efficient mechanisms for the reconstruction of the hyperspectral images from their compressive measurements. In particular, we considered and compared four different type of estimation algorithms: OMP, GPSR, LASSO, and IST. Furthermore, the large dimensions of hyperspectral images required the implementation of a practical block CASSI model to reconstruct the images with an acceptable delay and affordable computational cost. In order to consider the particularities of the block model and the dispersive effects in the CASSI-like sensing procedure, the problem was reformulated, as well as the construction of the variables involved. For this practical CASSI setup, we evaluated the performance of the overall system by considering the aforementioned algorithms and the different factors that impacted the reconstruction procedure. Finally, the obtained results were analyzed and discussed from a practical perspective.

Deep-learning-based hyperspectral imaging through a RGB camera

Hyperspectral image (HSI) contains both spatial pattern and spectral information which has been widely used in food safety, remote sensing, and medical detection. However, the acquisition of hyperspectral images is usually costly due to the complicated apparatus for the acquisition of optical spectrum. Recently, it has been reported that HSI can be reconstructed from single RGB image using convolution neural network (CNN) algorithms. Compared with the traditional hyperspectral cameras, the method based on CNN algorithms is simple, portable and low cost. In this study, we focused on the influence of the RGB camera spectral sensitivity (CSS) on the HSI. A Xenon lamp incorporated with a monochromator were used as the standard light source to calibrate the CSS. And the experimental results show that the CSS plays a significant role in the reconstruction accuracy of an HSI. In addition, we proposed a new HSI reconstruction network where the dimensional structure of the original hyperspectral datacube was modified by 3D matrix transpose to improve the reconstruction accuracy.

Classification Learning of Latent Bruise Damage to Apples Using Shortwave Infrared Hyperspectral Imaging.

Bruise damage is a very commonly occurring defect in apple fruit which facilitates disease occurrence and spread, leads to fruit deterioration and can greatly contribute to postharvest loss. The detection of bruises at their earliest stage of development can be advantageous for screening purposes. An experiment to induce soft bruises in Golden Delicious apples was conducted by applying impact energy at different levels, which allowed to investigate the detectability of bruises at their latent stage. The existence of bruises that were rather invisible to the naked eye and to a digital camera was proven by reconstruction of hyperspectral images of bruised apples, based on effective wavelengths and data dimensionality reduced hyperspectrograms. Machine learning classifiers, namely ensemble subspace discriminant (ESD), k-nearest neighbors (KNN), support vector machine (SVM) and linear discriminant analysis (LDA) were used to build models for detecting bruises at their latent stage, to study the influence of time after bruise occurrence on detection performance and to model quantitative aspects of bruises (severity), spanning from latent to visible bruises. Over all classifiers, detection models had a higher performance than quantitative ones. Given its highest speed in prediction and high classification performance, SVM was rated most recommendable for detection tasks. However, ESD models had the highest classification accuracy in quantitative (>85%) models and were found to be relatively better suited for such a multiple category classification problem than the rest.

Hyperspectral Anomaly Detection Based on Subspace Low-Rank Decomposition

This paper mainly studies the anomaly detection algorithm on the basis of denoising and reconstruction of hyperspectral image, combined with some basic methods, such as subspace representation, tensor decomposition, spectral global and spatial non-local similar low-rank decomposition, norm constraint and so on. the mixed noise (Gaussian noise, impulse noise and dead line) of hyperspectral images which seriously affect the accuracy of anomaly detection are preprocessed. On this basis, the global RX algorithm is used to detect anomalies in the denoised hyperspectral image, and the simulation data are compared with the original real data. The experimental results show that the subspace low-rank decomposition anomaly detection algorithm is better than other existing algorithms in speed and accuracy, which shows the feasibility and superiority of this algorithm.

End-to-end joint spectral–spatial compression and reconstruction of hyperspectral images using a 3D convolutional autoencoder

Hyperspectral images (HSIs) must be preprocessed by a compression model, which reduces the pressure of storing and transmitting huge data in applications. Whereas most of the existing methods consider only the compression or reconstruction requirements, an end-to-end optimization would simultaneously improve the performance of both requirements. We propose a three-dimensional convolutional autoencoder that precisely achieves end-to-end joint spectral–spatial compression and reconstruction of HSIs. In an experimental evaluation, the proposed method improved the spectral angle mapper, peak signal-to-noise ratio, and structural similarity index measurement of the reconstructed HSIs by 20.8% to 33.1%, 0.9% to 11.5%, and 0.5% to 2.2%, respectively, relative to competitive methods.

Hyperspectral image sparse reconstruction model based on collaborative multidimensional correlation

In recent years, the sparse reconstruction of hyperspectral images based on compressed sensing theory has attracted great attention of the research community. How to better explore the sparsity priori that conforms to the statistical properties and structural properties of the original hyperspectral images is one of the key problems. To address this, in this paper, a novel hyperspectral image sparse reconstruction model based on collaborative multidimensional correlations is proposed. First, considering the high-dimensional and high-redundancy properties of hyperspectral data, the correlations of hyperspectral images are studied from multiple dimensions, except the “spectral correlation” between hyperspectral bands and the “local spatial correlation” within a single band, the “overall nonlocal correlation” is novel defined in band groups with highly correlated spectral features, where the “nonlocal correlation” of the key band in the band group can be used to guide the “nonlocal correlation” of other bands within the group. Then, the sparse measurement models are provided corresponding to the above correlations in different dimensions, at the same time, a band group collaborative sparse model which integrates the multidimensional correlations has been proposed. Finally, the proposed band group collaborative sparse model is used as the sparse constraint to construct the hyperspectral sparse reconstruction framework. Experimental results demonstrate that, the newly defined “global non-local correlation” can effectively enhance the detail information in the reconstruction process of hyperspectral images, which provides the possibility for better analyzing land-cover objects in the subsequent applications.

Spectral dimensional correlation and sparse reconstruction model of hyperspectral images

The high dimensionality and redundancy properties of hyperspectral image (HSI) constitute great challenges for transmission and process. Recently, HSI reconstruction based on compressed sensing has received increasing attention and has become a frontier problem. Effectively exploiting the sparse prior of HSI is crucial to improve the reconstruction quality. In this paper, the spectral dimension plane of the HSI band group is defined, and the structural correlation of HSI spectral dimension is studied. The following conclusions can be obtained. First, the texture distribution of the HSI spectral dimension is simpler and more uniform than that of the HIS spatial domain, and its texture exhibits low contrast and high properties, thus simplifying the sparse representation process. Second, in the HSI spectral dimension plane, the search area adjacent to the reference block has a certain number of spectral curve blocks that are highly similar to the reference block. Based on this observation, the structure correlation of the HIS spectral dimension is defined, and a sparse measurement model is proposed. Finally, the sparse model S-SCoSM is proposed by integrating the sparse representation of spatial dimension nonlocal correlation and the spectral dimension structure correlation. The sparse reconstruction model of HSI is constructed using this sparse constrain prior. Experimental results show that by further exploring the correlation of HSI from the viewpoint of spatial and spectral domains, the proposed sparse model S-SCoSM obtains a more adequate and effective HSI sparse constraint prior; hence, the reconstruction quality of HSI is improved. Consequently, the spatial information quality of the reconstructed band image can be effectively improved as well as the spectral attributes of band groups can be well maintained.

M2H-Net: A Reconstruction Method For Hyperspectral Remotely Sensed Imagery

Hyperspectral remote sensing can get spatially and spectrally continuous data simultaneously. However, the imaging equipment is usually expensive and complex, along with the low spatial resolution. In recent years, reconstruction of hyperspectral image by deep learning from the widely used low-cost, high spatial resolution RGB camera, has attracted extensive attention in many fields. However, most research is limited to three bands in the range of 400–700 nm, which greatly restrains its application in remote sensing. In this study, a more suitable for remote sensing multispectral to hyperspectral network (M2H-Net) is proposed, which can take many bands as input and output hyperspectral images with any number of bands within a wider spectral range (380–2500 nm). Its characteristics include adding residual connection on U-Net to reduce vanishing gradients; adding convolution combinations with different kernel sizes (1 × 1 and 3 × 3) to balance the spectral and spatial relationships. It is applied on images from different platforms (UAVs and Satellites), different imaging modes (frame and pushbroom) and different spectral response functions (narrow and wide bandwidth), and the results show that: 1) it has a very high accuracy of hyperspectral image reconstruction. The mean relative absolute error (MRAE) and root mean squared error (RMSE) are between 0.039 and 0.074 and 0.010–0.016, respectively, which are 69.2% and 41.2% lower than those of U-Net; 2) it has high efficiency with fast convergence (about 40 epochs) and stable performance. Compared with many algorithms won in the new trends in image restoration and enhancement (NTIRE) competition, M2H-Net ranked 7th in accuracy, but took less time (0.44 s); 3) it has strong generalization ability. Using the pre-trained M2H-Nets to reconstruct Cubert S185 and GF-5 hyperspectral images in different locations, different times and complex scenes, high accuracy (MRAE = 0.072, RMSE = 0.011) can still be obtained. This method is more suitable for remote sensing to meet the needs of multiple bands, spectrum width and complex scenes, thus provides the possibility to generate the global coverage hyperspectral imagery by using the massive in-orbit or historical archived multispectral images, which will not only greatly save the R&D and investment on hyperspectral imaging equipment, but also conduct data collection with higher efficiency and lower complexity. Due to the ability to reconstruct hyperspectral images in specified bands on demand, M2H-Net is also of great value in hyperspectral image processing, such as data compression, storage and transmission, etc.

Double Ghost Convolution Attention Mechanism Network: A Framework for Hyperspectral Reconstruction of a Single RGB Image.

Current research on the reconstruction of hyperspectral images from RGB images using deep learning mainly focuses on learning complex mappings through deeper and wider convolutional neural networks (CNNs). However, the reconstruction accuracy of the hyperspectral image is not high and among other issues the model for generating these images takes up too much storage space. In this study, we propose the double ghost convolution attention mechanism network (DGCAMN) framework for the reconstruction of a single RGB image to improve the accuracy of spectral reconstruction and reduce the storage occupied by the model. The proposed DGCAMN consists of a double ghost residual attention block (DGRAB) module and optimal nonlocal block (ONB). DGRAB module uses GhostNet and PRELU activation functions to reduce the calculation parameters of the data and reduce the storage size of the generative model. At the same time, the proposed double output feature Convolutional Block Attention Module (DOFCBAM) is used to capture the texture details on the feature map to maximize the content of the reconstructed hyperspectral image. In the proposed ONB, the Argmax activation function is used to obtain the region with the most abundant feature information and maximize the most useful feature parameters. This helps to improve the accuracy of spectral reconstruction. These contributions enable the DGCAMN framework to achieve the highest spectral accuracy with minimal storage consumption. The proposed method has been applied to the NTIRE 2020 dataset. Experimental results show that the proposed DGCAMN method outperforms the spectral accuracy reconstructed by advanced deep learning methods and greatly reduces storage consumption.

Fast Hyperspectral Image Recovery of Dual-Camera Compressive Hyperspectral Imaging via Non-Iterative Subspace-Based Fusion.

Coded aperture snapshot spectral imaging (CASSI) is a promising technique for capturing three-dimensional hyperspectral images (HSIs), in which algorithms are used to perform the inverse problem of HSI reconstruction from a single coded two-dimensional (2D) measurement. Due to the ill-posed nature of this problem, various regularizers have been exploited to reconstruct 3D data from 2D measurements. Unfortunately, the accuracy and computational complexity are unsatisfactory. One feasible solution is to utilize additional information such as the RGB measurement in CASSI. Considering the combined CASSI and RGB measurements, in this paper, we propose a fusion model for HSI reconstruction. Specifically, we investigate the low-dimensional spectral subspace property of HSIs composed of a spectral basis and spatial coefficients. In particular, the RGB measurement is utilized to estimate the coefficients, while the CASSI measurement is adopted to provide the spectral basis. We further propose a patch processing strategy to enhance the spectral low-rank property of HSIs. The optimization of the proposed model requires neither iteration nor the spectral sensing matrix of the RGB detector. Extensive experiments on both simulated and real HSI datasets demonstrate that our proposed method not only outperforms previous state-of-the-art (iterative algorithms) methods in quality but also speeds up the reconstruction by more than 5000 times.

基于改进残差密集网络的高光谱重建

基于改进残差密集网络的高光谱重建

Coded Hyperspectral Image Reconstruction Using Deep External and Internal Learning.

To solve the low spatial and/or temporal resolution problem which the conventional hyperspectral cameras often suffer from, coded hyperspectral imaging systems have attracted more attention recently. Recovering a hyperspectral image (HSI) from its corresponding coded image is an ill-posed inverse problem, and learning accurate prior of HSI is essential to solve this inverse problem. In this paper, we present an effective convolutional neural network (CNN) based method for coded HSI reconstruction, which learns the deep prior from the external dataset as well as the internal information of input coded image with spatial-spectral constraint. Specifically, we first develop a CNN-based channel attention reconstruction network to effectively exploit the spatial-spectral correlation of the HSI. Then, the reconstruction network is learned by leveraging an arbitrary external hyperspectral dataset to exploit the general spatial-spectral correlation under adversarial loss. Finally, we customize the network by internal learning with spatial-spectral constraint and total variation regularization for each coded image, which can make use of the internal imaging model to learn specific prior for current desirable image and effectively avoids overfitting. Experimental results using both synthetic data and real images show that our method outperforms the state-of-the-art methods on several popular coded hyperspectral imaging systems under both comprehensive quantitative metrics and perceptive quality.

Single pixel hyperspectral Cherenkov-excited fluorescence imaging with LINAC X-ray sheet scanning and spectral unmixing.

Cherenkov light induced from megavolt (MV) X-rays during external beam radiotherapy serves as an internal light source to excite phosphors or fluorophores within biological tissues for molecular imaging. The broad spectrum of Cherenkov light leads to significant spectral overlap with any luminescence emission and, to overcome this problem, a single pixel hyperspectral imaging methodology was demonstrated here by coupling the detection with light sheet scanning and filtered back projection reconstruction of hyperspectral images. Thin scanned sheets of MV X-rays produce Cherenkov light to illuminate the planes deep within the tissue-simulating media. A fluorescence probe was excited by Cherenkov light, and a complete hyperspectral sinogram of the data was obtained through translation and rotation of the beam. Hyperspectral 2D images finally were reconstructed. Through this approach of spectral unmixing, it was possible to resolve hyperspectral images of both the Cherenkov and resulting fluorescence intensity from molecular sensors.

Hyperspectral image super-resolution through clustering-based sparse representation

Promoting the spatial resolution of hyperspectral sensors is expected to improve computer vision tasks. However, due to the physical limitations of imaging sensors, the hyperspectral image is often of low spatial resolution. In this paper, we propose a new hyperspectral image super-resolution method from a low-resolution (LR) hyperspectral image and a high resolution (HR) multispectral image of the same scene. The reconstruction of HR hyperspectral image is formulated as a joint estimation of the hyperspectral dictionary and the sparse codes based on the spatial-spectral sparsity of the hyperspectral image. The hyperspectral dictionary is learned from the LR hyperspectral image. The sparse codes with respect to the learned dictionary are estimated from LR hyperspectral image and the corresponding HR multispectral image. To improve the accuracy, both spectral dictionary learning and sparse coefficients estimation exploit the spatial correlation of the HR hyperspectral image. Experiments show that the proposed method outperforms several state-of-art hyperspectral image super-resolution methods in objective quality metrics and visual performance.