Low Resolution Hyperspectral Image Research Articles

Advancing Hyperspectral and Multispectral Image Fusion: An Information-Aware Transformer-Based Unfolding Network.

In hyperspectral image (HSI) processing, the fusion of the high-resolution multispectral image (HR-MSI) and the low-resolution HSI (LR-HSI) on the same scene, known as MSI-HSI fusion, is a crucial step in obtaining the desired high-resolution HSI (HR-HSI). With the powerful representation ability, convolutional neural network (CNN)-based deep unfolding methods have demonstrated promising performances. However, limited receptive fields of CNN often lead to inaccurate long-range spatial features, and inherent input and output images for each stage in unfolding networks restrict the feature transmission, thus limiting the overall performance. To this end, we propose a novel and efficient information-aware transformer-based unfolding network (ITU-Net) to model the long-range dependencies and transfer more information across the stages. Specifically, we employ a customized transformer block to learn representations from both the spatial and frequency domains as well as avoid the quadratic complexity with respect to the input length. For spatial feature extractions, we develop an information transfer guided linearized attention (ITLA), which transmits high-throughput information between adjacent stages and extracts contextual features along the spatial dimension in linear complexity. Moreover, we introduce frequency domain learning in the feedforward network (FFN) to capture token variations of the image and narrow the frequency gap. Via integrating our proposed transformer blocks with the unfolding framework, our ITU-Net achieves state-of-the-art (SOTA) performance on both synthetic and real hyperspectral datasets.

A hyperspectral surgical microscope with super-resolution reconstruction for intraoperative image guidance.

Hyperspectral imaging (HSI) is an emerging imaging modality in medical applications, especially for intraoperative image guidance. A surgical microscope improves surgeons' visualization with fine details during surgery. The combination of HSI and surgical microscope can provide a powerful tool for surgical guidance. However, to acquire high-resolution hyperspectral images, the long integration time and large image file size can be a burden for intraoperative applications. Super-resolution reconstruction allows acquisition of low-resolution hyperspectral images and generates high-resolution HSI. In this work, we developed a hyperspectral surgical microscope and employed our unsupervised super-resolution neural network, which generated high-resolution hyperspectral images with fine textures and spectral characteristics of tissues. The proposed method can reduce the acquisition time and save storage space taken up by hyperspectral images without compromising image quality, which will facilitate the adaptation of hyperspectral imaging technology in intraoperative image guidance.

S2CycleDiff: Spatial-Spectral-Bilateral Cycle-Diffusion Framework for Hyperspectral Image Super-resolution

Hyperspectral image super-resolution (HISR) is a technique that can break through the limitation of imaging mechanism to obtain the hyperspectral image (HSI) with high spatial resolution. Although some progress has been achieved by existing methods, most of them directly learn the spatial-spectral joint mapping between the observed images and the target high-resolution HSI (HrHSI), failing to fully reserve the spectral distribution of low-resolution HSI (LrHSI) and the spatial distribution of high-resolution multispectral imagery (HrMSI). To this end, we propose a spatial-spectral-bilateral cycle-diffusion framework (S2CycleDiff) for HISR, which can step-wise generate the HrHSI with high spatial-spectral fidelity by learning the conditional distribution of spatial and spectral super-resolution processes bilaterally. Specifically, a customized conditional cycle-diffusion framework is designed as the backbone to achieve the spatial-spectral-bilateral super-resolution by repeated refinement, wherein the spatial/spectral guided pyramid denoising (SGPD) module seperately takes HrMSI and LrHSI as the guiding factors to achieve the spatial details injection and spectral correction. The outputs of the conditional cycle-diffusion framework are fed into a complementary fusion block to integrate the spatial and spectral details to generate the desired HrHSI. Experiments have been conducted on three widely used datasets to demonstrate the superiority of the proposed method over state-of-the-art HISR methods. The code is available at https://github.com/Jiahuiqu/S2CycleDiff.

Spatial and Spectral Reconstruction of Breast Lumpectomy Hyperspectral Images.

(1) Background: Hyperspectral imaging has emerged as a promising margin assessment technique for breast-conserving surgery. However, to be implicated intraoperatively, it should be both fast and capable of yielding high-quality images to provide accurate guidance and decision-making throughout the surgery. As there exists a trade-off between image quality and data acquisition time, higher resolution images come at the cost of longer acquisition times and vice versa. (2) Methods: Therefore, in this study, we introduce a deep learning spatial-spectral reconstruction framework to obtain a high-resolution hyperspectral image from a low-resolution hyperspectral image combined with a high-resolution RGB image as input. (3) Results: Using the framework, we demonstrate the ability to perform a fast data acquisition during surgery while maintaining a high image quality, even in complex scenarios where challenges arise, such as blur due to motion artifacts, dead pixels on the camera sensor, noise from the sensor's reduced sensitivity at spectral extremities, and specular reflections caused by smooth surface areas of the tissue. (4) Conclusion: This gives the opportunity to facilitate an accurate margin assessment through intraoperative hyperspectral imaging.

Hyperspectral and multispectral image fusion via residual selective kernel attention-based U-net

ABSTRACT The fusion of low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI) is a crucial technology for producing high-resolution hyperspectral images. Most existing image fusion algorithms based on deep learning do not fully utilize the ability of neural network to extract and process multi-scale features, which leads to the problem of difficulty in fully learning features and ambiguity of features. In order to overcome these issues, a residual selective kernel attention-based U-net named RSKAU-net is designed for LR-HSI and HR-MSI fusion. RSKAU-net is constructed by a residual selective kernel module with an attention mechanism and a channel attention block. The residual selective kernel attention-based (RSKA) module is designed to process images of different resolutions, which adaptively extracts multi-scale features and efficiently emphasizes significant features through the attention mechanism. The channel attention (CA) module retains important spectral information by assigning different weights to each channel of LR-HSI. The proposed network can enhance the spatial information of LR-HSI while preserving its spectral information. Meanwhile, it effectively fuses the features from the source image to obtain the HR-HSI with rich details. The experimental results demonstrate that the proposed network has advantages in terms of both visual effect and objective quantitative indices when compared to existing HSI-MSI fusion approaches.

Unsupervised Deep Tensor Network for Hyperspectral-Multispectral Image Fusion.

Fusing low-resolution (LR) hyperspectral images (HSIs) with high-resolution (HR) multispectral images (MSIs) is a significant technology to enhance the resolution of HSIs. Despite the encouraging results from deep learning (DL) in HSI-MSI fusion, there are still some issues. First, the HSI is a multidimensional signal, and the representability of current DL networks for multidimensional features has not been thoroughly investigated. Second, most DL HSI-MSI fusion networks need HR HSI ground truth for training, but it is often unavailable in reality. In this study, we integrate tensor theory with DL and propose an unsupervised deep tensor network (UDTN) for HSI-MSI fusion. We first propose a tensor filtering layer prototype and further build a coupled tensor filtering module. It jointly represents the LR HSI and HR MSI as several features revealing the principal components of spectral and spatial modes and a sharing code tensor describing the interaction among different modes. Specifically, the features on different modes are represented by the learnable filters of tensor filtering layers, the sharing code tensor is learned by a projection module, in which a co-attention is proposed to encode the LR HSI and HR MSI and then project them onto the sharing code tensor. The coupled tensor filtering module and projection module are jointly trained from the LR HSI and HR MSI in an unsupervised and end-to-end way. The latent HR HSI is inferred with the sharing code tensor, the features on spatial modes of HR MSIs, and the spectral mode of LR HSIs. Experiments on simulated and real remote-sensing datasets demonstrate the effectiveness of the proposed method.

Hyperspectral Image Super Resolution With Real Unaligned RGB Guidance.

Fusion-based hyperspectral image (HSI) super-resolution has become increasingly prevalent for its capability to integrate high-frequency spatial information from the paired high-resolution (HR) RGB reference (Ref-RGB) image. However, most of the existing methods either heavily rely on the accurate alignment between low-resolution (LR) HSIs and RGB images or can only deal with simulated unaligned RGB images generated by rigid geometric transformations, which weakens their effectiveness for real scenes. In this article, we explore the fusion-based HSI super-resolution with real Ref-RGB images that have both rigid and nonrigid misalignments. To properly address the limitations of existing methods for unaligned reference images, we propose an HSI fusion network (HSIFN) with heterogeneous feature extractions, multistage feature alignments, and attentive feature fusion. Specifically, our network first transforms the input HSI and RGB images into two sets of multiscale features with an HSI encoder and an RGB encoder, respectively. The features of Ref-RGB images are then processed by a multistage alignment module to explicitly align the features of Ref-RGB with the LR HSI. Finally, the aligned features of Ref-RGB are further adjusted by an adaptive attention module to focus more on discriminative regions before sending them to the fusion decoder to generate the reconstructed HR HSI. Additionally, we collect a real-world HSI fusion dataset, consisting of paired HSI and unaligned Ref-RGB, to support the evaluation of the proposed model for real scenes. Extensive experiments are conducted on both simulated and our real-world datasets, and it shows that our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison. The code and dataset are publicly available at https://zeqiang-lai.github.io/HSI-RefSR/.

Progressive Spatial Information-Guided Deep Aggregation Convolutional Network for Hyperspectral Spectral Super-Resolution.

Fusion-based spectral super-resolution aims to yield a high-resolution hyperspectral image (HR-HSI) by integrating the available high-resolution multispectral image (HR-MSI) with the corresponding low-resolution hyperspectral image (LR-HSI). With the prosperity of deep convolutional neural networks, plentiful fusion methods have made breakthroughs in reconstruction performance promotions. Nevertheless, due to inadequate and improper utilization of cross-modality information, the most current state-of-the-art (SOTA) fusion-based methods cannot produce very satisfactory recovery quality and only yield desired results with a small upsampling scale, thus affecting the practical applications. In this article, we propose a novel progressive spatial information-guided deep aggregation convolutional neural network (SIGnet) for enhancing the performance of hyperspectral image (HSI) spectral super-resolution (SSR), which is decorated through several dense residual channel affinity learning (DRCA) blocks cooperating with a spatial-guided propagation (SGP) module as the backbone. Specifically, the DRCA block consists of an encoding part and a decoding part connected by a channel affinity propagation (CAP) module and several cross-layer skip connections. In detail, the CAP module is customized by exploiting the channel affinity matrix to model correlations among channels of the feature maps for aggregating the channel-wise interdependencies of the middle layers, thereby further boosting the reconstruction accuracy. Additionally, to efficiently utilize the two cross-modality information, we developed an innovative SGP module equipped with a simulation of the degradation part and a deformable adaptive fusion part, which is capable of refining the coarse HSI feature maps at pixel-level progressively. Extensive experimental results demonstrate the superiority of our proposed SIGnet over several SOTA fusion-based algorithms.

SSTF-Unet: Spatial-Spectral Transformer-Based U-Net for High-Resolution Hyperspectral Image Acquisition.

To obtain a high-resolution hyperspectral image (HR-HSI), fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) is a prominent approach. Numerous approaches based on convolutional neural networks (CNNs) have been presented for hyperspectral image (HSI) and multispectral image (MSI) fusion. Nevertheless, these CNN-based methods may ignore the global relevant features from the input image due to the geometric limitations of convolutional kernels. To obtain more accurate fusion results, we provide a spatial-spectral transformer-based U-net (SSTF-Unet). Our SSTF-Unet can capture the association between distant features and explore the intrinsic information of images. More specifically, we use the spatial transformer block (SATB) and spectral transformer block (SETB) to calculate the spatial and spectral self-attention, respectively. Then, SATB and SETB are connected in parallel to form the spatial-spectral fusion block (SSFB). Inspired by the U-net architecture, we build up our SSTF-Unet through stacking several SSFBs for multiscale spatial-spectral feature fusion. Experimental results on public HSI datasets demonstrate that the designed SSTF-Unet achieves better performance than other existing HSI and MSI fusion approaches.

Unsupervised Test-Time Adaptation Learning for Effective Hyperspectral Image Super-Resolution With Unknown Degeneration.

Fusing a low-resolution hyperspectral image (HSI) with a high-resolution (HR) multi-spectral image has provided an effective way for HSI super-resolution (SR). The key lies on inferring the posteriori of the latent (i.e., HR) HSI using an appropriate image prior and the likelihood determined by the degeneration between the latent HSI and the observed images. However, in scenarios with complex imaging environments and various imaging scenes, the prior of HSIs can be prohibitively complicated and the degeneration is often unknown, which causes it difficult to accurately infer the posteriori of each latent HSI. To tackle this problem, we present an unsupervised test-time adaptation learning (UTAL) framework for HSI SR under unknown degeneration. Instead of directly modeling the complicated image prior, it first implicitly learns a content-agnostic prior shared across different images through supervisedly pre-training a mutual-guiding fusion module on extensive synthetic data. Then, it adapts the shared prior to those private characteristics in the latent HSI for posteriori inference through unsupervisedly learning a self-guiding adaptation module and a degeneration estimation network on two observed images in the test phase. Such a two-stage learning scheme models the complicated image prior in a divide-and-conquer manner, which eases the modeling difficulty and improves the prior accuracy. Moreover, the unknown degeneration can be estimated properly. Both of these two advantages empower us to accurately infer the posteriori of the latent HSI, thereby increasing the generalization performance in real applications. Additionally, in order to further mitigate the over-fitting in coping with more challenging cases (e.g., degenerations in both spectral and spatial domains are unknown) and speed up, we propose to meta-train UTAL on extensive synthetic SR tasks and solve it using an alternative optimization strategy such that UTAL learns to produce good generalization performance in real challenging cases with a small number of gradient descent steps. To verify the efficacy of UTAL, we evaluate it on HSI SR tasks with different unknown degenerations as well as some other HSI restoration tasks (e.g., compressive sensing), and report strong results superior to that of existing competitors.

Reciprocal transformer for hyperspectral and multispectral image fusion

Hyperspectral and multispectral (HS–MS) image fusion aims to reconstruct high-resolution hyperspectral images from low-resolution hyperspectral images and high-resolution multispectral images. While convolutional neural networks have been widely used for HS–MS fusion, their potential is curtailed due to the limited receptive field of each neuron, resulting in inadequate long-range modeling capabilities. Although several Transformer-based HS–MS fusion methods have been proposed, most of them often fail to fully integrate and coordinate the data from the two modalities (i.e., hyperspectral images and multispectral images). Such ineffective interactions significantly compromise the quality of the reconstructed hyperspectral images. In this paper, we introduce a novel reciprocal fusion strategy called the dual cross Transformer-based fusion (DCTransformer) for HS–MS fusion. The model excels in capturing the interplay between different modalities by utilizing directional pairwise multi-head cross-attention, which facilitates the transfer of information between modalities. Additionally, we incorporate a Swin Transformer block post cross-attention to enhance the self-attention within the context. Extensive experiments show that our DCTransformer performs favorably against other recent works on both simulation HSI datasets and real HSI datasets. The source code and pre-trained models are availabe at https://github.com/qingma2016/DCTransformer.

Enhancing Feature Detection and Matching in Low-Pixel-Resolution Hyperspectral Images Using 3D Convolution-Based Siamese Networks.

Today, hyperspectral imaging plays an integral part in the remote sensing and precision agriculture field. Identifying the matching key points between hyperspectral images is an important step in tasks such as image registration, localization, object recognition, and object tracking. Low-pixel resolution hyperspectral imaging is a recent introduction to the field, bringing benefits such as lower cost and form factor compared to traditional systems. However, the use of limited pixel resolution challenges even state-of-the-art feature detection and matching methods, leading to difficulties in generating robust feature matches for images with repeated textures, low textures, low sharpness, and low contrast. Moreover, the use of narrower optics in these cameras adds to the challenges during the feature-matching stage, particularly for images captured during low-altitude flight missions. In order to enhance the robustness of feature detection and matching in low pixel resolution images, in this study we propose a novel approach utilizing 3D Convolution-based Siamese networks. Compared to state-of-the-art methods, this approach takes advantage of all the spectral information available in hyperspectral imaging in order to filter out incorrect matches and produce a robust set of matches. The proposed method initially generates feature matches through a combination of Phase Stretch Transformation-based edge detection and SIFT features. Subsequently, a 3D Convolution-based Siamese network is utilized to filter out inaccurate matches, producing a highly accurate set of feature matches. Evaluation of the proposed method demonstrates its superiority over state-of-the-art approaches in cases where they fail to produce feature matches. Additionally, it competes effectively with the other evaluated methods when generating feature matches in low-pixel resolution hyperspectral images. This research contributes to the advancement of low pixel resolution hyperspectral imaging techniques, and we believe it can specifically aid in mosaic generation of low pixel resolution hyperspectral images.

CMVFTA: Optimal regression and deep maxout with optimization algorithm for pan-sharpening

Pan-sharpening is a procedure to fuse the spatial detail of high-resolution multispectral images (HR-MSI) and low-resolution hyperspectral images (LR-HSI) to produce HR-MSI. Due to increase in high-resolution satellites, methods based on pan-sharpening are increasingly utilized all over the world. However, the majority of techniques consider pan-sharpening as a major issue, which hinders the discriminative ability. This work proposes an optimization-based deep model for pan-sharpening using LR-HSI and HR-MSI. Initially, the LR-HSI is input into an up-sampling mode, and the resulting image is fed into weighted linear regression. Concurrently, HR-MSI is supplied into weighted linear regression. Weighted linear regression is used to combine the upsampled LR-HSI and HR-MSI. The HR-MSI is then sent into the Deep Maxout network (DMN), which learns the priors via residual learning. Furthermore, the suggested Competitive Multi-Verse Feedback Artificial Tree (CMVFTA) strategy is used for DMN training, which is constructed by combining the Competitive Multi-Verse Optimizer (CMVO) and Feedback Artificial Tree (FAT) approaches. Finally, the DMN, LR-HSI, and HR-MSI outputs are merged together to provide a pan-sharpening image. The proposed CMVFTA-based DMN offered enhanced performance with Degree of Distortion (DD) of 0.0402 dB, Peak signal-to-noise ratio (PSNR) of 49.60 dB, Root Mean Squared Error (RMSE) of 0.330, Relative Average Spectral Error (RASE) of 0.322, Filtered Correlation Coefficients (FCC) of 0.874, Quality with no reference (QNR) of 76.19.

Parallel wavelet networks incorporating modality adaptation for hyperspectral image super-resolution

Hyperspectral images (HSIs) have been applied to a wide range of areas thanks to their high spectral resolutions. However, spatial resolutions of HSIs are inevitably compromised due to hardware limits. While HSIs can be enhanced by the super resolution techniques, there may be still spectral distortions and spatial blurriness in the reconstructed high-resolution HSIs (HR-HSIs). Therefore, this paper proposes a Parallel Framework based on Modality-Adaptive Wavelet Networks (PF-MAWN) to reconstruct HR-HSISs by effectively fusing low-resolution HSIs (LR-HSIs) and high-resolution multispectral images (HR-MSIs). To avoid spectral artifacts, multiple MAWNs work in parallel and fuse HR-MSIs/LR-HSIs in groups of highly correlated adjacent bands. Each MAWN incorporates wavelet transform (WT) deeply into convolutional neural networks (CNN) by layers of Multidomain Preprocessing Blocks (MPBs), Multimodal Adaptation Blocks (MABs) and Multilevel Connection Blocks (MCBs). Within each MAB, there are two Feature Alignment Modules (FAMs) based on self-attention to radiometrically match HR-MSIs/LR-HSIs and an Information Control Module (ICM) based on Gaussian-weighted attention to control textural details in HR-HSIs. The framework exhibits good extensibility and generalizability on spectral images of arbitrary bands and diversified scales. Experimental results show that PF-MAWN can reconstruct high-quality HR-HSIs with different sizes of objects, ranging from microscopic to macroscopic. It outperforms several state-of-the-art methods on four public datasets of histopathology tissues, everyday stuff and remote sensing objects.

Swin–MRDB: Pan-Sharpening Model Based on the Swin Transformer and Multi-Scale CNN

Pan-sharpening aims to create high-resolution spectrum images by fusing low-resolution hyperspectral (HS) images with high-resolution panchromatic (PAN) images. Inspired by the Swin transformer used in image classification tasks, this research constructs a three-stream pan-sharpening network based on the Swin transformer and a multi-scale feature extraction module. Unlike the traditional convolutional neural network (CNN) pan-sharpening model, we use the Swin transformer to establish global connections with the image and combine it with a multi-scale feature extraction module to extract local features of different sizes. The model combines the advantages of the Swin transformer and CNN, enabling fused images to maintain good local detail and global linkage by mitigating distortion in hyperspectral images. In order to verify the effectiveness of the method, this paper evaluates fused images with subjective visual and quantitative indicators. Experimental results show that the method proposed in this paper can better preserve the spatial and spectral information of images compared to the classical and latest models.

Thangka Hyperspectral Image Super-Resolution Based on a Spatial–Spectral Integration Network

Thangka refers to a form of Tibetan Buddhist painting on a fabric, scroll, or Thangka, often depicting deities, scenes, or mandalas. Deep-learning-based super-resolution techniques have been applied to improve the spatial resolution of hyperspectral images (HSIs), especially for the preservation and analysis of Thangka cultural heritage. However, existing CNN-based methods encounter difficulties in effectively preserving spatial information, due to challenges such as registration errors and spectral variability. To overcome these limitations, we present a novel cross-sensor super-resolution (SR) framework that utilizes high-resolution RGBs (HR-RGBs) to enhance the spectral features in low-resolution hyperspectral images (LR-HSIs). Our approach utilizes spatial–spectral integration (SSI) blocks and spatial–spectral restoration (SSR) blocks to effectively integrate and reconstruct spatial and spectral features. Furthermore, we introduce a frequency multi-head self-attention (F-MSA) mechanism that treats high-, medium-, and low-frequency features as tokens, enabling self-attention computations across the frequency dimension. We evaluate our method on a custom dataset of ancient Thangka paintings and demonstrate its effectiveness in enhancing the spectral resolution in high-resolution hyperspectral images (HR-HSIs), while preserving the spatial characteristics of Thangka artwork with minimal information loss.

RGB‐guided hyperspectral image super‐resolution with deep progressive learning

AbstractDue to hardware limitations, existing hyperspectral (HS) camera often suffer from low spatial/temporal resolution. Recently, it has been prevalent to super‐resolve a low resolution (LR) HS image into a high resolution (HR) HS image with a HR RGB (or multispectral) image guidance. Previous approaches for this guided super‐resolution task often model the intrinsic characteristic of the desired HR HS image using hand‐crafted priors. Recently, researchers pay more attention to deep learning methods with direct supervised or unsupervised learning, which exploit deep prior only from training dataset or testing data. In this article, an efficient convolutional neural network‐based method is presented to progressively super‐resolve HS image with RGB image guidance. Specifically, a progressive HS image super‐resolution network is proposed, which progressively super‐resolve the LR HS image with pixel shuffled HR RGB image guidance. Then, the super‐resolution network is progressively trained with supervised pre‐training and unsupervised adaption, where supervised pre‐training learns the general prior on training data and unsupervised adaptation generalises the general prior to specific prior for variant testing scenes. The proposed method can effectively exploit prior from training dataset and testing HS and RGB images with spectral‐spatial constraint. It has a good generalisation capability, especially for blind HS image super‐resolution. Comprehensive experimental results show that the proposed deep progressive learning method outperforms the existing state‐of‐the‐art methods for HS image super‐resolution in non‐blind and blind cases.

AOHSR: attention mechanism-based optical image-guided hyperspectral image super-resolution

ABSTRACT With the rapid development of hyperspectral imaging technology, hyperspectral image (HSI) has been applied in various fields. However, due to the constraint of hyperspectral imaging devices, the spatial resolution of HSI remains too low to accurately meet the needs of target detection and recognition, which shackles the further application of HSI. In this paper, an optical image-guided HSISR algorithm based on the attention mechanism (AOHSR) was proposed in order to effectively improve the resolution of HSI. With the help of the attention mechanism, the proposed algorithm can effectively retain the spectral information of HSI and the spatial information of the optical image. Even without high spatial resolution optical imaging in the test stage, favourable reconstruction results can still be achieved. Firstly, the framework of the proposed optical image-guided HSI SR algorithm based on the attention mechanism was introduced. The algorithm consists of three modules: spectral reservation module (SRM), optical image guidance module (OIGM) and spatial-spectral recovery module (SSRM). The SRM uses an improved channel attention mechanism to extract spectral features from low-resolution (LR) HSIs to preserve spectral information. Through an improved spatial attention mechanism, the OIGM extracts spatial information from high-resolution (HR) optical images, supplementing the information extracted from LR HSI. The SSRM is used to fuse the spectral information on HSI and the spatial information on optical images to reconstruct HR HSI. Then, a joint training algorithm for reconstruction loss and optical image content loss was also proposed, which can effectively guide to learn the reconstruction of detailed information of the spatial texture while ensuring the preservation of spectral information. The results of experiments on three open datasets reveal that the HSIs reconstructed by the proposed network performed better than the existing algorithms regardless of whether optical images in the test stage exist.

GuidedNet: A General CNN Fusion Framework via High-Resolution Guidance for Hyperspectral Image Super-Resolution

Hyperspectral image super-resolution (HISR) is about fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) to generate a high-resolution hyperspectral image (HR-HSI). Recently, convolutional neural network (CNN)-based techniques have been extensively investigated for HISR yielding competitive outcomes. However, existing CNN-based methods often require a huge amount of network parameters leading to a heavy computational burden, thus, limiting the generalization ability. In this article, we fully consider the characteristic of the HISR, proposing a general CNN fusion framework with high-resolution guidance, called GuidedNet. This framework consists of two branches, including 1) the high-resolution guidance branch (HGB) that can decompose the high-resolution guidance image into several scales and 2) the feature reconstruction branch (FRB) that takes the low-resolution image and the multiscaled high-resolution guidance images from the HGB to reconstruct the high-resolution fused image. GuidedNet can effectively predict the high-resolution residual details that are added to the upsampled HSI to simultaneously improve spatial quality and preserve spectral information. The proposed framework is implemented using recursive and progressive strategies, which can promote high performance with a significant network parameter reduction, even ensuring network stability by supervising several intermediate outputs. Additionally, the proposed approach is also suitable for other resolution enhancement tasks, such as remote sensing pansharpening and single-image super-resolution (SISR). Extensive experiments on simulated and real datasets demonstrate that the proposed framework generates state-of-the-art outcomes for several applications (i.e., HISR, pansharpening, and SISR). Finally, an ablation study and more discussions assessing, for example, the network generalization, the low computational cost, and the fewer network parameters, are provided to the readers. The code link is: https://github.com/Evangelion09/GuidedNet.

HMFT: Hyperspectral and Multispectral Image Fusion Super-Resolution Method Based on Efficient Transformer and Spatial-Spectral Attention Mechanism.

Due to the imaging mechanism of hyperspectral images, the spatial resolution of the resulting images is low. An effective method to solve this problem is to fuse the low-resolution hyperspectral image (LR-HSI) with the high-resolution multispectral image (HR-MSI) to generate the high-resolution hyperspectral image (HR-HSI). Currently, the state-of-the-art fusion approach is based on convolutional neural networks (CNN), and few have attempted to use Transformer, which shows impressive performance on advanced vision tasks. In this paper, a simple and efficient hybrid architecture network based on Transformer is proposed to solve the hyperspectral image fusion super-resolution problem. We use the clever combination of convolution and Transformer as the backbone network to fully extract spatial-spectral information by taking advantage of the local and global concerns of both. In order to pay more attention to the information features such as high-frequency information conducive to HR-HSI reconstruction and explore the correlation between spectra, the convolutional attention mechanism is used to further refine the extracted features in spatial and spectral dimensions, respectively. In addition, considering that the resolution of HSI is usually large, we use the feature split module (FSM) to replace the self-attention computation method of the native Transformer to reduce the computational complexity and storage scale of the model and greatly improve the efficiency of model training. Many experiments show that the proposed network architecture achieves the best qualitative and quantitative performance compared with the latest HSI super-resolution methods.