Spatial Resolution Hyperspectral Image Research Articles

Multispectral and hyperspectral images fusion based on subspace representation and nonlocal low-rank regularization

ABSTRACT Multispectral image (MSI) and hyperspectral image (HSI) fusion is a popular method for HSI super-resolution reconstruction (HSI-SR). MSI-HSI fusion problem is ill-posed and demands several image priors or regularization terms to solve accurately, which is a challenging issue. In this paper, we propose an MSI-HSI fusion model via subspace representation and nonlocal low-rank regularization (SRNLRR). SRNLRR model incorporates the global spectral correlations and spatial nonlocal similarities of HSI to improve the fusion results, where the priors complement each other. First, we use the mode-n tensor-matrix product to project latent high spatial resolution HSI (HR-HSI) into spectral subspace, which can capture spectral low-rankness and reduce computational complexity. Then, based on low-rank representation (LRR) and nonlocal processing strategy, we design a spatial nonlocal LRR regularization (spa-NLRR) and a spectral global LRR regularization (spe-GLRR). These two regularizations analyze the spatial nonlocal similarities and spectral global correlations from intermediate-level vision. Finally, we use the residual regularization program to obtain more image information and input it into the fusion model. We use alternate minimization (AM) methods to optimize the SRNLRR model and employ the alternating direction method (ADM) on spatial/spectral LRR learning. Comparison experiments between the SRNLRR method and six advanced methods on three HSI datasets indicate the superiority of our method.

A spectral and spatial transformer for hyperspectral remote sensing image super-resolution

ABSTRACT Due to the generally low spatial resolution of hyperspectral images (HSIs), early multispectral images lacked corresponding panchromatic bands, and as a result, fusion methods could not be used to enhance resolution. Many researchers have proposed various image super-resolution methods to address these limitations. However, these methods still suffered from issues, such as inadequate feature representation, lack of spectral feature representation, and high computational cost and inefficiency. To address these challenges, a spectral and spatial transformer (SST) algorithm for hyperspectral remote sensing image super-resolution is introduced. This algorithm uses a spatial transformer structure to extract the spatial features between the image pixels and a spectral transformer structure to extract the spectral features within the image pixels. The integration of these two components is applied to HSI super-resolution. After comparative experiments with currently advanced methods on three publicly available hyperspectral datasets, the results consistently show that our algorithm has better performance in both spectral fidelity and spatial restoration. Furthermore, our proposed algorithm was applied to real-world super-resolution experiments in the region of China's Ruoergai National Park, and subsequently, pixel-based classification was conducted on the super-resolution images, the results indicate that our algorithm could also be applied to future remote sensing interpretation tasks.

Hyperspectral Image Super-Resolution via Adaptive Factor Group Sparsity Regularization-Based Subspace Representation

Hyperspectral image (HSI) super-resolution is a vital technique that generates high spatial-resolution HSI (HR-HSI) by integrating information from low spatial-resolution HSI with high spatial-resolution multispectral image (MSI). However, existing subspace representation-based methods face challenges, including adaptive subspace dimension determination, inadequate spectral correlation capture, and expensive computation. In this paper, we propose a novel factor group sparsity regularized subspace representation (FGSSR)-based method for HSI super-resolution that can simultaneously address these issues encountered in previous methods. Specifically, by incorporating the factor group sparsity regularization into the subspace representation model, we first propose an FGSSR model to capture the spectral correlation property of the HR-HSI. The key advantage of FGSSR lies in its equivalence to the Schatten-p norm and its adaptive determination of the accurate subspace dimension, enabling it to capture spectral correlation more effectively. To preserve the spatial self-similarity prior in the HR-HSI, the tensor nuclear norm regularization on the low-dimensional coefficients is also incorporated into the proposed FGSSR-based model. Finally, an effective proximal alternating minimization-based algorithm is developed to solve the FGSSR-based model. Experimental results on the simulated and real datasets demonstrate that the proposed FGSSR-based method outperforms several state-of-the-art fusion methods with significant improvements.

Thangka Hyperspectral Image Super-Resolution Based on a Spatial–Spectral Integration Network

Thangka refers to a form of Tibetan Buddhist painting on a fabric, scroll, or Thangka, often depicting deities, scenes, or mandalas. Deep-learning-based super-resolution techniques have been applied to improve the spatial resolution of hyperspectral images (HSIs), especially for the preservation and analysis of Thangka cultural heritage. However, existing CNN-based methods encounter difficulties in effectively preserving spatial information, due to challenges such as registration errors and spectral variability. To overcome these limitations, we present a novel cross-sensor super-resolution (SR) framework that utilizes high-resolution RGBs (HR-RGBs) to enhance the spectral features in low-resolution hyperspectral images (LR-HSIs). Our approach utilizes spatial–spectral integration (SSI) blocks and spatial–spectral restoration (SSR) blocks to effectively integrate and reconstruct spatial and spectral features. Furthermore, we introduce a frequency multi-head self-attention (F-MSA) mechanism that treats high-, medium-, and low-frequency features as tokens, enabling self-attention computations across the frequency dimension. We evaluate our method on a custom dataset of ancient Thangka paintings and demonstrate its effectiveness in enhancing the spectral resolution in high-resolution hyperspectral images (HR-HSIs), while preserving the spatial characteristics of Thangka artwork with minimal information loss.

Hyperspectral Super-Resolution Reconstruction Network Based on Hybrid Convolution and Spectral Symmetry Preservation

Hyperspectral images (HSI) have high-dimensional and complex spectral characteristics, with dozens or even hundreds of bands covering the same area of pixels. The rich information of the ground objects makes hyperspectral images widely used in satellite remote sensing. Due to the limitations of remote sensing satellite sensors, hyperspectral images suffer from insufficient spatial resolution. Therefore, utilizing software algorithms to improve the spatial resolution of hyperspectral images has become an urgent problem that needs to be solved. The spatial information and spectral information of hyperspectral images are strongly correlated. If only the spatial resolution is improved, it often damages the spectral information. Inspired by the high correlation between spectral information in adjacent spectral bands of hyperspectral images, a hybrid convolution and spectral symmetry preservation network has been proposed for hyperspectral super-resolution reconstruction. This includes a model to integrate information from neighboring spectral bands to supplement target band feature information. The proposed model introduces flexible spatial-spectral symmetric 3D convolution in the network structure to extract low-resolution and neighboring band features. At the same time, a combination of deformable convolution and attention mechanisms is used to extract information from low-resolution bands. Finally, multiple bands are fused in the reconstruction module, and the high-resolution hyperspectral image containing global information is obtained by Fourier transform upsampling. Experiments were conducted on the indoor hyperspectral image dataset CAVE, the airborne hyperspectral dataset Pavia Center, and Chikusei. In the X2 super-resolution task, the PSNR values achieved on the CAVE, Pavia Center, and Chikusei datasets were 46.335, 36.321, and 46.310, respectively. In the X4 super-resolution task, the PSNR values achieved on the CAVE, Pavia Center, and Chikusei datasets were 41.218, 30.377, and 38.365, respectively. The results show that our method outperforms many advanced algorithms in objective indicators such as PSNR and SSIM while maintaining the spectral characteristics of hyperspectral images.

Hyperspectral Anomaly Detection Based on Regularized Background Abundance Tensor Decomposition

The low spatial resolution of hyperspectral images means that existing mixed pixels rely heavily on spectral information, making it difficult to differentiate between the target of interest and the background. The endmember extraction method is powerful in enhancing spatial structure information for hyperspectral anomaly detection, whereas most approaches are based on matrix representation, which inevitably destroys the spatial or spectral information. In this paper, we treated the hyperspectral image as a third-order tensor and proposed a novel anomaly detection method based on a low-rank linear mixing model of the scene background. The obtained abundance maps possessed more distinctive features than the raw data, which was beneficial for identifying anomalies in the background. Specifically, the low-rank tensor background was approximated as the mode-3 product of an abundance tensor and endmember matrix. Due to the distinctive features of the background’s abundance maps, we characterized them by tensor regularization and imposed low rankness through CP decomposition, smoothness, and sparsity. In addition, we utilized the ℓ1,1,2-norm to characterize the tube-wise sparsity of the anomaly, since it accounted for a small portion of the scene. The experimental results obtained using five real datasets demonstrated the outstanding performance of our proposed method.

Hyperspectral Image Super-resolution via Knowledge-Driven Deep Unrolling and Transformer Embedded Convolutional Recurrent Neural Network.

Hyperspectral (HS) imaging has been widely used in various real application problems. However, due to the hardware limitations, the obtained HS images usually have low spatial resolution, which could obviously degrade their performance. Through fusing a low spatial resolution HS image with a high spatial resolution auxiliary image (e.g., multispectral, RGB or panchromatic image), the so-called HS image fusion has underpinned much of recent progress in enhancing the spatial resolution of HS image. Nonetheless, a corresponding well registered auxiliary image cannot always be available in some real situations. To remedy this issue, we propose in this paper a newly single HS image super-resolution method based on a novel knowledge-driven deep unrolling technique. Precisely, we first propose a maximum a posterior based energy model with implicit priors, which can be solved by alternating optimization to determine an elementary iteration mechanism. We then unroll such iteration mechanism with an ingenious Transformer embedded convolutional recurrent neural network in which two structural designs are integrated. That is, the vision Transformer and 3D convolution learn the implicit spatial-spectral priors, and the recurrent hidden connections over iterations model the recurrence of the iterative reconstruction stages. Thus, an effective knowledge-driven, end-to-end and data-dependent HS image super-resolution framework can be successfully attained. Extensive experiments on three HS image datasets demonstrate the superiority of the proposed method over several state-of-the-art HS image super-resolution methods.

Generative Dual-Adversarial Network With Spectral Fidelity and Spatial Enhancement for Hyperspectral Pansharpening.

Hyperspectral (HS) pansharpening is of great importance in improving the spatial resolution of HS images for remote sensing tasks. HS image comprises abundant spectral contents, whereas panchromatic (PAN) image provides spatial information. HS pansharpening constitutes the possibility for providing the pansharpened image with both high spatial and spectral resolution. This article develops a specific pansharpening framework based on a generative dual-adversarial network (called PS-GDANet). Specifically, the pansharpening problem is formulated as a dual task that can be solved by a generative adversarial network (GAN) with two discriminators. The spatial discriminator forces the intensity component of the pansharpened image to be as consistent as possible with the PAN image, and the spectral discriminator helps to preserve spectral information of the original HS image. Instead of designing a deep network, PS-GDANet extends GANs to two discriminators and provides a high-resolution pansharpened image in a fraction of iterations. The experimental results demonstrate that PS-GDANet outperforms several widely accepted state-of-the-art pansharpening methods in terms of qualitative and quantitative assessment.

Hyperspectral image super-resolution using cluster-based deep convolutional networks

In recent years, deep convolutional neural networks (CNNs) have been widely exploited for the hyperspectral image (HSI) super-resolution and obtained remarkable performance. However, most of the existing CNN-based methods have two main problems. One is to use two-dimension (2D) convolution to extract spatial information without paying attention to the mining of spectral information of hyperspectral images. The other is to use three-dimension (3D) convolution, which reduces the efficiency of the model when the network parameters increase. To address the above issues, we propose clustering deep residual neural network (CDRNN) for hyperspectral image super-resolution in this paper. The proposed CDRNN learns the complex, nonlinear mappings between low spatial resolution HSI and high spatial resolution HSI. At first, an unsupervised clustering method is used to divide a low spatial resolution HSI into several classes according to spectral correlation. Then, the spectrum-pairs from the classified low spatial resolution HSI and the corresponding high spatial resolution HSI are used to train the CDRNN to establish the nonlinear mapping for each class. Finally, we classify the given low spatial resolution HSI into the determined category and use the trained CDRNN to reconstruct the final high spatial resolution HSI from the classified low spatial resolution HSI. We conduct extensive experiments on three simulated benchmark datasets and a real HSI to evaluate the super-resolution performance of the proposed method. Experimental results show that our proposed method achieves significant improvement over state-of-the-art methods.

A 3-Stage Spectral-Spatial Method for Hyperspectral Image Classification

Hyperspectral images often have hundreds of spectral bands of different wavelengths captured by aircraft or satellites that record land coverage. Identifying detailed classes of pixels becomes feasible due to the enhancement in spectral and spatial resolution of hyperspectral images. In this work, we propose a novel framework that utilizes both spatial and spectral information for classifying pixels in hyperspectral images. The method consists of three stages. In the first stage, the pre-processing stage, the Nested Sliding Window algorithm is used to reconstruct the original data by enhancing the consistency of neighboring pixels and then Principal Component Analysis is used to reduce the dimension of data. In the second stage, Support Vector Machines are trained to estimate the pixel-wise probability map of each class using the spectral information from the images. Finally, a smoothed total variation model is applied to ensure spatial connectivity in the classification map by smoothing the class probability tensor. We demonstrate the superiority of our method against three state-of-the-art algorithms on six benchmark hyperspectral datasets with 10 to 50 training labels for each class. The results show that our method gives the overall best performance in accuracy even with a very small set of labeled pixels. Especially, the gain in accuracy with respect to other state-of-the-art algorithms increases when the number of labeled pixels decreases, and, therefore, our method is more advantageous to be applied to problems with a small training set. Hence, it is of great practical significance since expert annotations are often expensive and difficult to collect.

Hyperspectral image super-resolution based on attention ConvBiLSTM network

ABSTRACT In this paper, a hyperspectral (HS) image super-resolution (SR) approach based on attention convolutional bi-long short-term memory (ConvBiLSTM) network is proposed, aiming to explore the collaborative spatial and spectral attention characteristics, thereby enhancing the spatial resolution of HS image. ConvBiLSTM network combines the spatial feature mining and sequential predicting abilities of convolutional neural network and recurrent neural network, respectively. We adapt the ConvBiLSTM network for our super-resolution purpose by regarding each band as a single frame of sequential data, and propose a band-sharing spatial-channel attention-combined ConvBiLSTM SR method to intensify the saliency features. Moreover, a spatial-regularized loss function is presented to further promote the fidelity of the super-resolved HS image. Experiments on four HS data sets show that the proposed approach outperforms some state-of-the-art HS image SR techniques, from the aspect of spectral fidelity.

A model-guided deep convolutional sparse coding network for hyperspectral and multispectral image fusion

ABSTRACT Although high-resolution hyperspectral images are of great significance in various applications, the manner in which to acquire such images remains a fundamental problem due to sensors’ physical limitations. Fusing high-spatial resolution multispectral images (HR-MSIs) with low-spatial resolution hyperspectral images (LR-HSIs) has become a common approach to increase the spatial resolution of hyperspectral images (HSIs). Various fusion methods have been developed to obtain images with high resolution in both the spatial and spectral domains; the main drawbacks of these methods are time consumption and spectral distortion. With the intention of developing a fast yet accurate image fusion algorithm, this paper is attempted to incorporate the benefits of deep learning and convolutional sparse coding to create a new fusion method. First, the inputs are initialized by solving the Sylvester equation from an up-sampled LR-HSI and an HR-MSI. Second, the learned iterative shrinkage thresholding algorithm is extended to its convolutional version as the central part of the model and recursively learn the sparse priors. Finally, a fused image is achieved by adding the residual to the initialized image. The proposed method is conducted on the widely used datasets (Harvard and iCVL). The experimental results are demonstrated that the proposed approach outperforms several state-of-the-art approaches in terms of quantitative and qualitative perspectives.

An Unsupervised Laplacian Pyramid Network for Radiometrically Accurate Data Fusion of Hyperspectral and Multispectral Imagery

Improving the spatial resolution of hyperspectral images (HSIs) has traditionally been an important topic in the field of remote sensing. Many approaches have been proposed based on various theories, including component substitution, multiresolution analysis, spectral unmixing, Bayesian probability, and tensor representation. However, these methods have some common disadvantages such that their performance degrades dramatically as the up-scale ratio increases, and they have little concern for the per-pixel radiometric accuracy of the sharpened image. Moreover, many learning-based methods have been proposed through decades of innovations, but most of them require a large set of training pairs, which is unpractical for many real problems. To solve these problems, we propose a stable hyperspectral sharpening method based on the Laplacian pyramid and the generative convolutional neural network (CNN), which achieves superior radiometric accuracy of the sharpened data in different up-scale ratios based on one single input pair. First, with a low-resolution HSI (LR-HSI) and high-resolution multispectral image (HR-MSI) pair, the preliminary high-resolution HSI (HR-HSI) is calculated via linear regression. Then, the high-frequency details of the preliminary HR-HSI are estimated via the subtraction between it and the CNN-generated-blurry version. By injecting the details to the output of the generative CNN with the LR-HSI as input, the final HR-HSI is obtained. Nine different state-of-the-art sharpening methods are chosen as our baselines, and three different datasets with different scene content are tested. Furthermore, the target detection method, the adaptive coherence estimator (ACE), is conducted on the reconstructed HR-HSI to evaluate the per-pixel radiometric accuracy. The results demonstrate that the proposed method has the best and the most stable performance in terms of spectral and spatial accuracies.

3-D Auxiliary Classifier GAN for Hyperspectral Anomaly Detection via Weakly Supervised Learning

Hyperspectral anomaly detection (AD) is important in Earth observation and remote sensing. However, the low spatial resolution of hyperspectral images, insufficient samples and lack of prior information limit the detection accuracy. To solve these problems, in this paper, we propose an auxiliary classifier generative adversarial network model based on a three-dimensional (3D) convolutional neural network named 3D AC-GAN. Firstly, the model is based on a 3D convolutional neural network design, with 3D tensors as samples. The network maintains valuable image spatial spectrum joint features to achieve good detection results. It can also generate sufficient samples to achieve dataset augmentation, solving the overfitting problem in GAN training. Secondly, we train the model with a weakly supervised method. The label of the samples is obtained through the coarse scanning method. Then, the AC-GAN is trained with the bootstrapping method to mitigate the impact of noise labels. The experimental results show that our proposed algorithm outperforms state-of-the-art AD algorithms.

Unmixing-Based PAN-Guided Fusion Network for Hyperspectral Imagery

The hyperspectral image (HSI) has been widely used in many applications due to its fruitful spectral information. However, the limitation of imaging sensors has reduced its spatial resolution that causes detail loss. One solution is to fuse the low spatial resolution HSI (LR-HSI) and the panchromatic (PAN) image with inverse features to get the high-resolution HSI (HR-HSI). Most of the existing fusion methods just focus on small fusion ratios like 4 or 6, which might be impractical for some large ratios’ HSI and PAN image pairs. Moreover, the ill-posedness of restoring detail information in HSI with hundreds of bands from PAN image with only one band has not been solved effectively, especially under large fusion ratios. Therefore, a lightweight unmixing-based PAN-guided fusion network (Pgnet) is proposed to mitigate this ill-posedness and improve the fusion performance significantly. Note that the fusion process of the proposed network is under the projected low-dimensional abundance subspace with an extremely large fusion ratio of 16. Furthermore, based on the linear and nonlinear relationships between the PAN intensity and abundance, an interpretable PAN detail inject network (PDIN) is designed to inject the PAN details into the abundance feature efficiently. Comprehensive experiments on simulated and real datasets demonstrate the superiority and generality of our method over several state-of-the-art (SOTA) methods qualitatively and quantitatively (the codes in pytorch and paddle versions and dataset could be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/rs-lsl/Pgnet</uri> ).

Learning Based Super Resolution Application for Hyperspectral Images

Due to its spectral properties, hyperspectral imaging is superior to other types of imaging tools in identifying, distinguishing and classifying objects. Hyperspectral imaging instruments can detect light reflected from certain wavelengths between infrared and ultraviolet, apart from the wavelength that the human eye can distinguish on the electromagnetic spectrum. While this feature provides detailed information about the spectral feature of the object under investigation, it causes its spatial resolution to be low due to the technical overlap between spatial resolution and spectral resolution. Today, applications of hyperspectral images are increasing in important fields such as agriculture, mining, medicine and pharmacy, especially for military purposes. In order for applications to produce more precise results, high spatial resolution is required, as well as high spectral information. Hardware solving of low spatial resolution problem is a difficult and costly method. Therefore, software solution is an interesting area in the field of image processing. In this thesis, a hybrid solution method based on deep learning and sparse representation is proposed to increase the low spatial resolution of hyperspectral images. The method obtains a super-resolution image from a single hyperspectral image with a low spatial image with a deep convolutional mesh. Later, the super-resolution image obtained and the original low-spatial-resolution hyperspectral image are fused with the dictionary learning method, resulting in a new super-resolution image with high spectral and spatial resolutions. The application results show that our method achieves successful results compared to many super resolution applications in the literature.

Deep-learning-based latent space encoding for spectral unmixing of geological materials

The relatively coarse spatial resolution of hyperspectral images causes the mixing of disparate materials’ spectral responses in the sensor's instantaneous field of view (IFOV), resulting in mixed pixels. The current study proposes a capsule-based generative encoding model, called a denoising unmixing encoder network (DUENet), to formulate an end-to-end trainable spectral unmixing model. The reconstruction and cross-entropy losses and input prior-based constraints achieve joint optimization of denoising, data imputation, and spectral unmixing. Unlike earlier approaches, interpolation-based convolution and dynamic time wrapping (DTW)-based convolutional units facilitate DUENet to even unmix noisy spectra. In addition to embedding label information to improve the physical significance of the latent space, DUENet dynamically learns the parameters of the interpolation kernels. Benchmark airborne hyperspectral datasets (the Nabesna and Cuprite datasets) and simulated datasets were employed to evaluate the performance of the proposed approach. It was observed that the proposed joint optimization of spectral unmixing and denoising significantly improves the results. The adopted feature characterization using capsules improves the generalizability and gives good results, even with a limited number of training samples. This study shows the need for interpretability-based evaluation measures to analyze the unmixing frameworks based on the concepts learned for each endmember. The experiments confirm that the proposed strategy significantly reduces the models’ sensitivity to network parameters.

PoNet: A universal physical optimization-based spectral super-resolution network for arbitrary multispectral images

Spectral super-resolution is a very important technique to obtain hyperspectral images from only multispectral images, which can effectively solve the high acquisition cost and low spatial resolution of hyperspectral images. However, in practice, multispectral channels or images captured by the same sensor are often with different spatial resolutions, which brings a severe challenge to spectral super-resolution. This paper proposed a universal spectral super-resolution network based on physical optimization unfolding for arbitrary multispectral images, including single-resolution and cross-scale multispectral images. Furthermore, two new strategies are proposed to make full use of the spectral information, namely, cross-dimensional channel attention and cross-depth feature fusion. Experimental results on five data sets show superiority and stability of PoNet addressing any spectral super-resolution situations.

A Hardware Accelerator for Onboard Spatial Resolution Enhancement of Hyperspectral Images

Hyperspectral images (HSIs) are the images that have a high spectral resolution with hundreds of bands. HSIs are widely used in the precise identification and classification of the materials and surfaces. However, they have a low spatial resolution caused by the limited capacity of the sensors responsible for capturing the images. In view of this, the preprocessing techniques are used to increase the spatial resolution of the HSIs and enable greater precision in the subsequent processing stages. This letter presents the development of a hardware accelerator specially designed to process a pansharpening algorithm that fuses the hyperspectral and panchromatic images to produce a high spatial resolution HSI. The experimental results demonstrate that the processor is energy-efficient and can be used for onboard processing in a small spacecraft, such as the satellites.

Correction of Sunglint Effects in High Spatial Resolution Hyperspectral Imagery Using SWIR or NIR Bands and Taking Account of Spectral Variation of Refractive Index of Water

Passive high spatial resolution hyperspectral and multispectral imaging systems in the solar spectral region from aircraft and satellite platforms are being increasingly used for remote sensing of coastal waters and inland lakes. However, the remotely sensed data are often contaminated by the specular reflection of solar radiation at the air/water interface. Thus, the effects of sunglint need to be corrected. The purpose of sunglint correction is to remove the undesired specular reflected portion of radiance that does not carry any information about the water body and its bottom surface properties and to obtain the water-leaving radiance from the measured total radiance. In general, adequate treatment of the spectral variation of the sunglint effect with due consideration of the physics and the spectroscopy principles has not been done in previously reported hyperspectral sunglint removing techniques. In this study, a practical sunglint removing technique using either a shortwave infrared (SWIR) band or a near-infrared (NIR) band has been developed, where the liquid water absorption is sufficiently large, and the water-leaving radiance (in units of reflectance) is negligible. In the proposed method, proper consideration has been taken for the spectral Fresnel reflection in the wavelength range of 0.35–2.5 mm. Four cases of sunglint removal from hyperspectral imaging data acquired from two airborne imaging spectrometers have been presented.