Spectral Super-resolution Research Articles

Multistage Spatial-Spectral Fusion Network for Spectral Super-Resolution.

Spectral super-resolution (SSR) aims to restore a hyperspectral image (HSI) from a single RGB image, in which deep learning has shown impressive performance. However, the majority of the existing deep-learning-based SSR methods inadequately address the modeling of spatial-spectral features in HSI. That is to say, they only sufficiently capture either the spatial correlations or the spectral self-similarity, which results in a loss of discriminative spatial-spectral features and hence limits the fidelity of the reconstructed HSI. To solve this issue, we propose a novel SSR network dubbed multistage spatial-spectral fusion network (MSFN). From the perspective of network design, we build a multistage Unet-like architecture that differentially captures the multiscale features of HSI both spatialwisely and spectralwisely. It consists of two types of the self-attention mechanism, which enables the proposed network to achieve global modeling of HSI comprehensively. From the perspective of feature alignment, we innovatively design the spatial fusion module (SpatialFM) and spectral fusion module (SpectralFM), aiming to preserve the comprehensively captured spatial correlations and spectral self-similarity. In this manner, the multiscale features can be better fused and the accuracy of reconstructed HSI can be significantly enhanced. Quantitative and qualitative experiments on the two largest SSR datasets (i.e., NTIRE2022 and NTIRE2020) demonstrate that our MSFN outperforms the state-of-the-art SSR methods. The code implementation will be uploaded at https://github.com/Matsuri247/MSFN-for-Spectral-Super-Resolution.

DLRA-Net: Deep Local Residual Attention Network with Contextual Refinement for Spectral Super-Resolution

DLRA-Net: Deep Local Residual Attention Network with Contextual Refinement for Spectral Super-Resolution

Spectral Super-Resolution for High-Accuracy Rice Variety Classification using Hybrid CNN-Transformer Model

Rice variety classification is crucial for ensuring the purity and quality of rice production. In this study, hyperspectral images (HSI) of rice varieties were reconstructed using a spectral super-resolution technique, serving as a basis for an enhanced classification process. RGB images of various rice varieties were utilized to reconstruct the HSI, and a hybrid CNN-Transformer model featuring an efficient feature fusion module aimed at reducing redundancy was developed. Building on the spectral super-resolution model, a Swin-Transformer model was employed for rice variety classification, chosen for its capacity to effectively handle high-dimensional data with fewer parameters and lower FLOPs. The classification accuracy based on actual HSI reached an impressive 99.961%, while the accuracy based on reconstructed HSI was a close 99.413%. These results demonstrate that the spectral super-resolution method not only effectively reconstructs HSI images of rice varieties but also supports highly reliable classification. It is indicated by the study that spectral super-resolution can significantly outperform advanced CNN and Transformer-based methods in terms of accuracy and computational efficiency, offering a promising approach for precise rice variety classification and potentially other agricultural applications.

HPRN: Holistic Prior-Embedded Relation Network for Spectral Super-Resolution.

Spectral super-resolution (SSR) refers to the hyperspectral image (HSI) recovery from an RGB counterpart. Due to the one-to-many nature of the SSR problem, a single RGB image can be reprojected to many HSIs. The key to tackle this ill-posed problem is to plug into multisource prior information such as the natural spatial context prior of RGB images, deep feature prior, or inherent statistical prior of HSIs so as to effectively alleviate the degree of ill-posedness. However, most current approaches only consider the general and limited priors in their customized convolutional neural networks (CNNs), which leads to the inability to guarantee the confidence and fidelity of reconstructed spectra. In this article, we propose a novel holistic prior-embedded relation network (HPRN) to integrate comprehensive priors to regularize and optimize the solution space of SSR. Basically, the core framework is delicately assembled by several multiresidual relation blocks (MRBs) that fully facilitate the transmission and utilization of the low-frequency content prior of RGBs. Innovatively, the semantic prior of RGB inputs is introduced to mark category attributes, and a semantic-driven spatial relation module (SSRM) is invented to perform the feature aggregation of clustered similar ranges for refining recovered characteristics. In addition, we develop a transformer-based channel relation module (TCRM), which breaks the habit of employing scalars as the descriptors of channelwise relations in the previous deep feature prior and replaces them with certain vectors to make the mapping function more robust and smoother. In order to maintain the mathematical correlation and spectral consistency between hyperspectral bands, the second-order prior constraints (SOPCs) are incorporated into the loss function to guide the HSI reconstruction. Finally, extensive experimental results on four benchmarks demonstrate that our HPRN can reach the state-of-the-art performance for SSR quantitatively and qualitatively. Furthermore, the effectiveness and usefulness of the reconstructed spectra are verified by the classification results on the remote sensing dataset. Codes are available at https://github.com/Deep-imagelab/HPRN.

Spectral Super-Resolution via Model-Guided Cross-Fusion Network.

Spectral super-resolution, which reconstructs a hyperspectral image (HSI) from a single red-green-blue (RGB) image, has acquired more and more attention. Recently, convolution neural networks (CNNs) have achieved promising performance. However, they often fail to simultaneously exploit the imaging model of the spectral super-resolution and complex spatial and spectral characteristics of the HSI. To tackle the above problems, we build a novel cross fusion (CF)-based model-guided network (called SSRNet) for spectral super-resolution. In specific, based on the imaging model, we unfold the spectral super-resolution into the HSI prior learning (HPL) module and imaging model guiding (IMG) module. Instead of just modeling one kind of image prior, the HPL module is composed of two subnetworks with different structures, which can effectively learn the complex spatial and spectral priors of the HSI, respectively. Furthermore, a CF strategy is used to establish the connection between the two subnetworks, which further improves the learning performance of the CNN. The IMG module results in solving a strong convex optimization problem, which adaptively optimizes and merges the two features learned by the HPL module by exploiting the imaging model. The two modules are alternately connected to achieve optimal HSI reconstruction performance. Experiments on both the simulated and real data demonstrate that the proposed method can achieve superior spectral reconstruction results with relatively small model size. The code will be available at https://github.com/renweidian.

Multi-scale hyperspectral recovery networks: RGB-hyperspectral imaging consistency empowered deep spectral super-resolution.

Hyperspectral imaging, which analyzes a spectrum of light instead of primary colors, can provide rich physical and chemical information not regularly available to traditional imaging modalities. To physically obtain hyperspectral images, various methods have been designed relying on dispersion elements, filters, and coded masks, inevitably making the system complicated and expensive. To address these challenges, emerging deep neural networks can achieve spectral super-resolution by tackling the ill-posed problem, thereby enhancing the spectral recovery capabilities of RGB cameras. However, most of them ignore the consistency in spatial structure between the RGB and hyperspectral images, which can provide additional physical prior knowledge for spectral super-resolution tasks. To increase the fidelity of hyperspectral recovery from RGB images, we propose multi-scale hyperspectral recovery networks (MHRNs), designed to take full consideration of the consistency. In addition, physical constraints are employed to train MHRNs for better performance and interpretability. The spectral super-resolution applicability of the MHRN on publicly available datasets as well as practical applications suggests that it has great potential for advancing spectral recovery across various practical applications such as remote sensing, medical imaging, agricultural testing, and industrial inspection.

Generating high-resolution hyperspectral time series datasets based on unsupervised spatial-temporal-spectral fusion network incorporating a deep prior

Over the past decade, image fusion has emerged as an indispensable tool for surface monitoring due to its capability to reconstruct high-quality surface reflectance. While satisfactory progress has been made in the spatial-spectral and spatial-temporal fusion of remote sensing images, the fusion of spatial-temporal-spectral data remains a challenging task. This challenge arises from the presence of multiple nonlinear relationships caused by inconsistent temporal, spatial, and spectral resolutions. In this paper, we propose a novel approach to address the spatial-temporal-spectral fusion of remote sensing images using a triple discriminator Generative Adversarial Network called TDGAN. Our approach represents the first attempt at spatial-temporal-spectral fusion based on deep learning techniques. Specifically, TDGAN achieves unsupervised learning by incorporating a deep spectral transformation prior, explores the application performance of the Swin Transformer in spectral super-resolution for the first time. Additionally, three discriminators are designed based on the prior consistencies: phenological consistency for data acquired at different times and spatial-spectral consistency for data acquired at the same time. These discriminators effectively enhance the prediction accuracy of missing hyperspectral images. Extensive simulations and real experiments were conducted to evaluate the effectiveness of our proposed TDGAN. The results demonstrate its robustness and comparable performance to other state-of-the-art methods. The proposed approach provides high-quality hyperspectral datasets with high spatial and temporal resolutions, facilitating continuous land cover observations.

Spectral Superresolution Using Transformer with Convolutional Spectral Self-Attention

Hyperspectral images (HSI) find extensive application across numerous domains of study. Spectral superresolution (SSR) refers to reconstructing HSIs from readily available RGB images using the mapping relationships between RGB images and HSIs. In recent years, convolutional neural networks (CNNs) have become widely adopted in SSR research, primarily because of their exceptional ability to extract features. However, most current CNN-based algorithms are weak in terms of extracting the spectral features of HSIs. While certain algorithms can reconstruct HSIs through the fusion of spectral and spatial data, their practical effectiveness is hindered by their substantial computational complexity. In light of these challenges, we propose a lightweight network, Transformer with convolutional spectral self-attention (TCSSA), for SSR. TCSSA comprises a CNN-Transformer encoder and a CNN-Transformer decoder, in which the convolutional spectral self-attention blocks (CSSABs) are the basic modules. Multiple cascaded encoding and decoding modules within TCSSA facilitate the efficient extraction of spatial and spectral contextual information from HSIs. The convolutional spectral self-attention (CSSA) as the basic unit of CSSAB combines CNN with self-attention in the transformer, effectively extracting both spatial local features and global spectral features from HSIs. Experimental validation of TCSSA’s effectiveness is performed on three distinct datasets: GF5 for remote sensing images along with CAVE and NTIRE2022 for natural images. The experimental results demonstrate that the proposed method achieves a harmonious balance between reconstruction performance and computational complexity.

A novel spectral super-resolution network with dominant information between spatial and spectral domains

Existing spectral super-resolution (SSR) methods have achieved satisfactory performance by designing complicated deep convolution neural networks (DCNNs) to extract spectral and spatial features. However, these methods ignore the fact that the significance of spatial and spectral information in each hyperspectral image (HSI) is different, and most of them directly fuse two kinds of information with concatenation and convolution operation, which resulting in generating redundant information and have negative effects on reconstruction. To address such inadequacies, this paper proposes a novel adaptive spatial–spectral modulation network (ASSM-Net). Specifically, we propose a new adaptive feature fusion module (AFFM) to replace traditional convolutional fusion schemes. Through explicitly measuring the weights of spatial information and spectral features of HSI, AFFM can select dominant features for each pixel to modulate spatial and spectral information. Additionally, we develop a pixel-weighted aware attention (PAA) mechanism to enhance the feature interdependences for recovering finer structure information. Finally, a large number of quantitative and qualitative experiments reveal that the proposed network achieves competitive reconstruction results on three benchmark datasets (NTIRE 2022, CAVE and Harvard).

Separation-free spectral super-resolution via convex optimization

Atomic norm methods have recently been proposed for spectral super-resolution with flexibility in dealing with missing data and miscellaneous noises. A notorious drawback of these convex optimization methods however is their lower resolution in the high signal-to-noise (SNR) regime as compared to conventional methods such as ESPRIT. In this paper, we devise a simple weighting scheme in existing atomic norm methods and show that in theory the resolution of the resulting convex optimization method can be made arbitrarily high in the absence of noise, achieving the so-called separation-free super-resolution. This is proved by a novel, kernel-free construction of the dual certificate whose existence guarantees exact super-resolution using the proposed method. Numerical results corroborating our analysis are provided.

Tensor-Based Sparse Representation for Hyperspectral Image Reconstruction Using RGB Inputs

Hyperspectral image (HSI) reconstruction from RGB input has drawn much attention recently and plays a crucial role in further vision tasks. However, current sparse coding algorithms often take each single pixel as the basic processing unit during the reconstruction process, which ignores the strong similarity and relation between adjacent pixels within an image or scene, leading to an inadequate learning of spectral and spatial features in the target hyperspectral domain. In this paper, a novel tensor-based sparse coding method is proposed to integrate both spectral and spatial information represented in tensor forms, which is capable of taking all the neighboring pixels into account during the spectral super-resolution (SSR) process without breaking the semantic structures, thus improving the accuracy of the final results. Specifically, the proposed method recovers the unknown HSI signals using sparse coding on the learned dictionary pairs. Firstly, the spatial information of pixels is used to constrain the sparse reconstruction process, which effectively improves the spectral reconstruction accuracy of pixels. In addition, the traditional two-dimensional dictionary learning is further extended to the tensor domain, by which the structure of inputs can be processed in a more flexible way, thus enhancing the spatial contextual relations. To this end, a rudimentary HSI estimation acquired in the sparse reconstruction stage is further enhanced by introducing the regression method, aiming to eliminate the spectral distortion to some extent. Abundant experiments are conducted on two public datasets, indicating the considerable availability of the proposed framework.

Spectral Super-Resolution in Frequency Domain.

Spectral super-resolution aims to reconstruct a hyperspectral image (HSI) from its corresponding RGB image, which has drawn much more attention in remote sensing field. Recent advances in the application of deep learning models for spectral super-resolution have demonstrated great potential. However, these methods only work in spectral-spatial domain while rarely explore the potential property in the frequency domain. In this work, we first attempt to address spectral super-resolution in the frequency domain. To well merge the frequency information into the super-resolution network, a spectral-spatial-frequency domain fusion network (SSFDF) is designed, which consists of three key parts: frequency-domain feature learning, spectral-spatial domain feature learning, and feature fusion module. In more detail, a frequency-domain feature learning network is first exploited to dig the frequency-domain information of the input data. Then, a symmetric convolutional neural network (CNN) is developed to acquire the spectral-spatial features of the input data, where a parameter-sharing strategy is utilized to reduce network parameters. Finally, a feature fusion module is proposed to reconstruct HSI. Comprehensive experiments on several datasets reveal that our method can attain state-of-the-art reconstruction result with respect to other spectral super-resolution techniques.

Spectral Super-Resolution via Deep Low-Rank Tensor Representation.

Spectral super-resolution has attracted the attention of more researchers for obtaining hyperspectral images (HSIs) in a simpler and cheaper way. Although many convolutional neural network (CNN)-based approaches have yielded impressive results, most of them ignore the low-rank prior of HSIs resulting in huge computational and storage costs. In addition, the ability of CNN-based methods to capture the correlation of global information is limited by the receptive field. To surmount the problem, we design a novel low-rank tensor reconstruction network (LTRN) for spectral super-resolution. Specifically, we treat the features of HSIs as 3-D tensors with low-rank properties due to their spectral similarity and spatial sparsity. Then, we combine canonical-polyadic (CP) decomposition with neural networks to design an adaptive low-rank prior learning (ALPL) module that enables feature learning in a 1-D space. In this module, there are two core modules: the adaptive vector learning (AVL) module and the multidimensionwise multihead self-attention (MMSA) module. The AVL module is designed to compress an HSI into a 1-D space by using a vector to represent its information. The MMSA module is introduced to improve the ability to capture the long-range dependencies in the row, column, and spectral dimensions, respectively. Finally, our LTRN, mainly cascaded by several ALPL modules and feedforward networks (FFNs), achieves high-quality spectral super-resolution with fewer parameters. To test the effect of our method, we conduct experiments on two datasets: the CAVE dataset and the Harvard dataset. Experimental results show that our LTRN not only is as effective as state-of-the-art methods but also has fewer parameters. The code is available at https://github.com/renweidian/LTRN.

Progressive Spatial Information-Guided Deep Aggregation Convolutional Network for Hyperspectral Spectral Super-Resolution.

Fusion-based spectral super-resolution aims to yield a high-resolution hyperspectral image (HR-HSI) by integrating the available high-resolution multispectral image (HR-MSI) with the corresponding low-resolution hyperspectral image (LR-HSI). With the prosperity of deep convolutional neural networks, plentiful fusion methods have made breakthroughs in reconstruction performance promotions. Nevertheless, due to inadequate and improper utilization of cross-modality information, the most current state-of-the-art (SOTA) fusion-based methods cannot produce very satisfactory recovery quality and only yield desired results with a small upsampling scale, thus affecting the practical applications. In this article, we propose a novel progressive spatial information-guided deep aggregation convolutional neural network (SIGnet) for enhancing the performance of hyperspectral image (HSI) spectral super-resolution (SSR), which is decorated through several dense residual channel affinity learning (DRCA) blocks cooperating with a spatial-guided propagation (SGP) module as the backbone. Specifically, the DRCA block consists of an encoding part and a decoding part connected by a channel affinity propagation (CAP) module and several cross-layer skip connections. In detail, the CAP module is customized by exploiting the channel affinity matrix to model correlations among channels of the feature maps for aggregating the channel-wise interdependencies of the middle layers, thereby further boosting the reconstruction accuracy. Additionally, to efficiently utilize the two cross-modality information, we developed an innovative SGP module equipped with a simulation of the degradation part and a deformable adaptive fusion part, which is capable of refining the coarse HSI feature maps at pixel-level progressively. Extensive experimental results demonstrate the superiority of our proposed SIGnet over several SOTA fusion-based algorithms.

Spectral Super-Resolution by Using Universal and Private Jointed Spectral Library and Its Applications

Spectral Super-Resolution by Using Universal and Private Jointed Spectral Library and Its Applications

Spectral Library-Based Spectral Super-Resolution Under Incomplete Spectral Coverage Conditions

Spectral Library-Based Spectral Super-Resolution Under Incomplete Spectral Coverage Conditions

Frequency-Spatial Domain Feature Fusion for Spectral Super-Resolution

Frequency-Spatial Domain Feature Fusion for Spectral Super-Resolution

A General Deep Learning Point–Surface Fusion Framework for RGB Image Super-Resolution

Hyperspectral images are usually acquired in a scanning-based way, which can cause inconvenience in some situations. In these cases, RGB image spectral super-resolution technology emerges as an alternative. However, current mainstream spectral super-resolution methods aim to generate continuous spectral information at a very narrow range, limited to the visible light range. Some researchers introduce hyperspectral images as auxiliary data. But it is usually required that the auxiliary hyperspectral images have the same spatial range as RGB images. To address this issue, a general point–surface data fusion method is designed to achieve the RGB image spectral super-resolution goal in this paper, named GRSS-Net. The proposed method utilizes hyperspectral point data as auxiliary data to provide spectral reference information. Thus, the spectral super-resolution can extend the spectral reconstruction range according to spectral data. The proposed method utilizes compressed sensing theory as a fundamental physical mechanism and then unfolds the traditional hyperspectral image reconstruction optimization problem into a deep network. Finally, a high-spatial-resolution hyperspectral image can be obtained. Thus, the proposed method combines the non-linear feature extraction ability of deep learning and the interpretability of traditional physical models simultaneously. A series of experiments demonstrates that the proposed method can effectively reconstruct spectral information in RGB images. Meanwhile, the proposed method provides a framework of spectral super-resolution for different applications.

Extending function mixture network for improved spectral super-resolution

Hyperspectral Image reconstruction from RGB images is a low-cost and convenient alternative to acquiring hyperspectral images directly. The challenge in estimating the spectral response function and using it for generating the hyperspectral image data is addressed effectively by the use of convolutional neural networks for the task. The accuracy of the reconstruction techniques is improving and effective architectures are being adapted for the purpose. Specifically, in a recent work the authors propose the use of Function Mixture Network, to model the mapping from RGB to hyperspectral. This technique benefits from the use of an adaptive spatial receptive field, however, given that the goal is spectral superresolution additional use of adaptive spectral field is prudent. Accordingly, in this paper we propose an improved Function Mixture Network based reconstruction technique by additionally incorporating a variable spectral receptive field. The proposed updates have resulted in significant gains in performance as is demonstrated by the experiments conducted over multiple datasets. An additional update to the architecture is the use of non-convolutional branches in the function mixture network. The experiments of three standard datasets, clearly demonstrate the superiority of the proposed technique for hyperspectral image reconstruction based on RGB data.

Infrared spectral super-resolution model with linear canonical transforms regularization for spectral signals

Infrared spectral super-resolution has achieved great success for spectral signals under the noise-free case. However, the random noise and band overlap restricted the super-resolution performance in the real scenarios. Infrared spectral data acquisition is slow because weak signals (band overlap and random noise) limits its wide applications. To address this problem, a novel spectral super-resolution model is proposed with fast linear canonical transform (LCT) regularization, which can effectively improve the resolution of noisy and low-resolution spectrum. To reveal the sparsity difference between the observed infrared spectra and the real one, the LCT tool is utilized to distinguish the various coefficients distributions. The distribution of the real Infrared spectra is sparser than the observed infrared one, thus, an infrared spectral restoration model is proposed to regularize the sparsity distribution of the observed spectra by L0-norm. Experimental results on simulated and real spectral signals show that the proposed algorithm can effectively preserve the details of infrared spectrum and suppress random noise.