Hyperspectral Image Super-resolution Method Research Articles

Hyperspectral Image Super-Resolution Based on Spatial Correlation-Regularized Unmixing Convolutional Neural Network

Super-resolution (SR) technology has emerged as an effective tool for image analysis and interpretation. However, single hyperspectral (HS) image SR remains challenging, due to the high spectral dimensionality and lack of available high-resolution information of auxiliary sources. To fully exploit the spectral and spatial characteristics, in this paper, a novel single HS image SR approach is proposed based on a spatial correlation-regularized unmixing convolutional neural network (CNN). The proposed approach takes advantage of a CNN to explore the collaborative spatial and spectral information of an HS image and infer the high-resolution abundance maps, thereby reconstructing the anticipated high-resolution HS image via the linear spectral mixture model. Moreover, a dual-branch architecture network and spatial spread transform function are employed to characterize the spatial correlation between the high- and low-resolution HS images, aiming at promoting the fidelity of the super-resolved image. Experiments on three public remote sensing HS images demonstrate the feasibility and superiority in terms of spectral fidelity, compared with some state-of-the-art HS image super-resolution methods.

Hyperspectral Image Super-Resolution under the Guidance of Deep Gradient Information

Hyperspectral image (HSI) super-resolution has gained great attention in remote sensing, due to its effectiveness in enhancing the spatial information of the HSI while preserving the high spectral discriminative ability, without modifying the imagery hardware. In this paper, we proposed a novel HSI super-resolution method via a gradient-guided residual dense network (G-RDN), in which the spatial gradient is exploited to guide the super-resolution process. Specifically, there are three modules in the super-resolving process. Firstly, the spatial mapping between the low-resolution HSI and the desired high-resolution HSI is learned via a residual dense network. The residual dense network is used to fully exploit the hierarchical features learned from all the convolutional layers. Meanwhile, the gradient detail is extracted via a residual network (ResNet), which is further utilized to guide the super-resolution process. Finally, an empirical weight is set between the fully obtained global hierarchical features and the gradient details. Experimental results and the data analysis on three benchmark datasets with different scaling factors demonstrated that our proposed G-RDN achieved favorable performance.

Exploring the Relationship Between 2D/3D Convolution for Hyperspectral Image Super-Resolution

Hyperspectral image super-resolution (SR) methods based on deep learning have achieved significant progress recently. However, previous methods lack the joint analysis between spectrum and horizontal or vertical direction. Besides, when both 2D and 3D convolution are in the network, the existing models cannot effectively combine the two. To address these issues, in this article, we propose a novel hyperspectral image SR method by exploring the relationship between 2D/3D convolution (ERCSR). Our method alternately employs 2D and 3D units to solve the problem of structural redundancy by sharing spatial information during reconstruction for existing model, which can enhance the learning ability of 2D spatial domain. Importantly, compared with the network using 3D units, i.e., 2D units are replaced by 3D units, it can not only reduce the size of the model but also improve the performance of the model. Furthermore, to exploit the spectrum fully, the split adjacent spatial and spectral convolution (SAEC) is designed to parallelly explore information between spectrum and horizontal or vertical direction in space. Experiments on widely used benchmark datasets demonstrate that the proposed approach outperforms state-of-the-art SR algorithms across different scales in terms of quantitative and qualitative analysis.

A Spectral Grouping and Attention-Driven Residual Dense Network for Hyperspectral Image Super-Resolution

Although unprecedented success has been achieved in convolutional neural network (CNN)-based super-resolution (SR) for natural images, hyperspectral image (HSI) SR without auxiliary high-resolution images remains a challenging task due to the high spectral dimensionality, where learning effective spatial and spectral representations is of great importance. In this article, we introduce a novel CNN-based HSI SR method, termed spectral grouping and attention-driven residual dense network (SGARDN) to facilitate the modeling of all spectral bands and focus on the exploration of spatial-spectral features. Considering the block characteristic of HSI, we employ group convolutions in and between groups composed of highly similar spectral bands at early stages to extract informative spatial features and avoid spectral disorder caused by normal convolution. To exploit spectral prior, a new spectral attention mechanism constructed by covariance statistics of features is designed to adaptively recalibrate features. We adapt the spectral attention for group convolutions to rescale grouping features with holistic spectral information. These two sequential operations called spectral grouping and integration module aim to extract effective shallow spatial-spectral features that are reused in the following layers. On the other hand, the residual dense block can better deal with spatial-spectral features by experimental comparison and hence is combined with the spectral attention to form a new basic building block for powerful feature expression and spectral correlation learning. The experimental results on synthesized and real-scenario HSIs demonstrate the feasibility and superiority of the proposed method over other state-of-the-art methods.

Model-Guided Deep Hyperspectral Image Super-Resolution.

The trade-off between spatial and spectral resolution is one of the fundamental issues in hyperspectral images (HSI). Given the challenges of directly acquiring high-resolution hyperspectral images (HR-HSI), a compromised solution is to fuse a pair of images: one has high-resolution (HR) in the spatial domain but low-resolution (LR) in spectral-domain and the other vice versa. Model-based image fusion methods including pan-sharpening aim at reconstructing HR-HSI by solving manually designed objective functions. However, such hand-crafted prior often leads to inevitable performance degradation due to a lack of end-to-end optimization. Although several deep learning-based methods have been proposed for hyperspectral pan-sharpening, HR-HSI related domain knowledge has not been fully exploited, leaving room for further improvement. In this paper, we propose an iterative Hyperspectral Image Super-Resolution (HSISR) algorithm based on a deep HSI denoiser to leverage both domain knowledge likelihood and deep image prior. By taking the observation matrix of HSI into account during the end-to-end optimization, we show how to unfold an iterative HSISR algorithm into a novel model-guided deep convolutional network (MoG-DCN). The representation of the observation matrix by subnetworks also allows the unfolded deep HSISR network to work with different HSI situations, which enhances the flexibility of MoG-DCN. Extensive experimental results are reported to demonstrate that the proposed MoG-DCN outperforms several leading HSISR methods in terms of both implementation cost and visual quality. The code is available at https://see.xidian.edu.cn/faculty/wsdong/Projects/MoG-DCN.htm.

Hyperspectral Image Superresolution via Deep Structure and Texture Interfusion

Limited by the imagery sensors, hyperspectral image (HSI) is characterized by its high spectral resolution, but low spatial resolution. HSI superresolution (SR) aims at enhancing the spatial resolution through the postprocessing techniques. Inspired by the important observation that the structure and the texture exhibit different sensitivities to the spatial resolution, we propose an HSI SR method via a residual structure-texture dense network. Specifically, there are three modules that are designed in the SR process. First, the input HSI is decomposed into the texture cube and the structural cube via relative total variation. Second, the mapping between the texture-structure cube of the input HSI and that of the desired HSI is learnt via two parallel subnetworks. Finally, the deep learnt texture cube is interfused with the deep learnt structural cube to achieve the reconstructed HSI with high spatial resolution via a upsample module. Contrary to the existed methods that mainly focus on the entire scene, our approach opens a new way for superresolving the HSI from a structure and texture recovery aspect. Experimental results and data analysis on both indoor scenes and outdoor scenarios have demonstrated the effectiveness of the proposed method.

Nonnegative and Nonlocal Sparse Tensor Factorization-Based Hyperspectral Image Super-Resolution

Hyperspectral image (HSI) super-resolution refers to enhancing the spatial resolution of a 3-D image with many spectral bands (slices). It is a seriously ill-posed problem when the low-resolution (LR) HSI is the only input. It is better solved by fusing the LR HSI with a high-resolution (HR) multispectral image (MSI) for a 3-D image with both high spectral and spatial resolution. In this article, we propose a novel nonnegative and nonlocal 4-D tensor dictionary learning-based HSI super-resolution model using group-block sparsity. By grouping similar 3-D image cubes into clusters and then conduct super-resolution cluster by cluster using 4-D tensor structure, we not only preserve the structure but also achieve sparsity within the cluster due to the collection of similar cubes. We use 4-D tensor Tucker decomposition and impose nonnegative constraints on the dictionaries and group-block sparsity. Numerous experiments demonstrate that the proposed model outperforms many state-of-the-art HSI super-resolution methods.

Hyperspectral Image Superresolution Using Spectrum and Feature Context

Deep learning-based hyperspectral image superresolution methods have achieved great success recently. However, most methods utilize 2D or 3D convolution to explore features, and rarely combine the two types of convolution to design networks. Moreover, when the model only contains 3D convolution, almost all the methods take all the bands of hyperspectral image as input to analyze, which requires more memory footprint. To address these issues, we explore a new structure for hyperspectral image superresolution using spectrum and feature context. Inspired by the high similarity among adjacent bands, we design a dual-channel network through 2D and 3D convolution to jointly exploit the information from both single band and adjacent bands, which is different from previous works. Under the connection of depth split, it can effectively share spatial information so as to improve the learning ability of 2D spatial domain. Besides, our method introduces the features extracted from previous band, which contributes to the complementarity of information and simplifies the network structure. Through feature context fusion, it significantly enhances the performance of the algorithm. Extensive evaluations and comparisons on three public datasets demonstrate that our approach produces the state-of-the-art results over the existing approaches.

Hyperspectral image super-resolution through clustering-based sparse representation

Promoting the spatial resolution of hyperspectral sensors is expected to improve computer vision tasks. However, due to the physical limitations of imaging sensors, the hyperspectral image is often of low spatial resolution. In this paper, we propose a new hyperspectral image super-resolution method from a low-resolution (LR) hyperspectral image and a high resolution (HR) multispectral image of the same scene. The reconstruction of HR hyperspectral image is formulated as a joint estimation of the hyperspectral dictionary and the sparse codes based on the spatial-spectral sparsity of the hyperspectral image. The hyperspectral dictionary is learned from the LR hyperspectral image. The sparse codes with respect to the learned dictionary are estimated from LR hyperspectral image and the corresponding HR multispectral image. To improve the accuracy, both spectral dictionary learning and sparse coefficients estimation exploit the spatial correlation of the HR hyperspectral image. Experiments show that the proposed method outperforms several state-of-art hyperspectral image super-resolution methods in objective quality metrics and visual performance.

Superpixel Spectral Unmixing for Hyperspectral Image Superresolution Using a Coupled Encoder-Decoder Network

In this paper, we propose a novel hyperspectral image superresolution method based on superpixel spectral unmixing using a coupled encoder-decoder network. The hyperspectral image and multispectral images are fused to generate high-resolution hyperspectral images through the spectral unmixing framework with low-rank constraint. Specifically, the endmember and abundance information is extracted via a coupled encoder-decoder network integrating the priori for unmixing. The coupled network consists of two encoders and one shared decoder, where spectral information is preserved through the encoder. The multispectral image is clustered into superpixels to explore self-similarity, and then, the superpixels are unmixed to obtain an abundance matrix. By imposing a low-rank constraint on the abundance matrix, we further improve the superresolution performance. Experiments on the CAVE and Harvard datasets indicate that our superresolution method outperforms the other compared methods in terms of quantitative evaluation and visual quality.

Hyperspectral image super-resolution combining with deep learning and spectral unmixing

In recent years, hyperspectral image super-resolution has attracted the attention of many researchers and has become a hot topic in the field of computer vision. However, it is difficult to obtain high-resolution images due to imaging hardware devices. At present, many existing hyperspectral image super-resolution methods have not achieved good results. In this paper, we propose a hyperspectral image super-resolution method combining with deep residual convolutional neural network (DRCNN) and spectral unmixing. Firstly, the spatial resolution of the image is enhanced by learning a priori knowledge of natural images. The DRCNN reconstructs high spatial resolution hyperspectral images by concatenating multiple residual blocks, each containing two convolutional layers. Secondly, the spectral features of low-resolution and high-resolution hyperspectral images are linked by spectral unmixing. This approach aims to obtain the endmember matrix and the abundance matrix. The final reconstruction result is obtained by multiplying the endmember matrix and the abundance matrix. In addition, in order to improve the visual effect of the reconstructed image, the total variation regularity is used to impose constraints on the abundance matrix to enhance the relationship between the pixels. The experimental results of remote sensing data based on ground facts show that the proposed method has good performance and preserves spatial information and spectral information without the need for auxiliary images.

Deep Recursive Network for Hyperspectral Image Super-Resolution

Fusion based hyperspectral image (HSI) super-resolution method, which obtains a spatially high-resolution (HR) HSI by fusing a low-resolution (LR) HSI and an HR conventional image, has been a prevalent method for HSI super-resolution. One effective fusion based method is to cast HSI super-resolution into a unified optimization problem, where handcrafted priors such as sparse prior or low rank prior are always adopted to regularize the latent HR HSI to be optimized. However, these priors show limitations in generalizing to challenging cases due to the heuristic assumption on image statistics as well as the restricted expressiveness capacity of the shallow structure. Taking advantages of the powerful expression ability of deep learning based method, a new HSI super-resolution network is proposed which implicitly incorporates a deep structure as the regularizer/prior. Specifically, we reformulate the original unified optimization problem into three sub-optimization problems, one is related with the regularizer and the others are without. Thanks to the fact that the one related with the regularizer naturally equals to a denoising problem, a recursive residual network is proposed for this sub-optimization problem. In addition, we unfold the other sub-optimization problems into network representations, with which the original unified optimization problem can be represented into a fully end-to-end network. Experimental results shows the superiority of the proposed method for HSI super-resolution on three benchmark datasets.

Non-local blind hyperspectral image super-resolution via 4d sparse tensor factorization and low-rank

Hyperspectral image (HSI) super-resolution is a technique to improve the spatial resolution of a HSI for better visual perception and down stream applications. This is a very ill-posed inverse problem and is often solved by fusing the low-resolution (LR) HSI with a high-resolution (HR) multispectral image (MSI). It is more challenging for blind HSI super-resolution, i.e., when the spatial degradation operators are completely unknown. In this paper, we propose a novel sparse tensor factorization model for the task of blind HSI super-resolution using the spatial non-local self-similarity and spectral global correlation of HSIs. Image clustering method is employed to collect some similar 3D cubes of HSIs which can be formed as some 4D image clusters with high correlation. We conduct cluster wise computation to not only save computation time but also to introduce a non-local regularity originated from the redundancy of cubes. By using the sparsity of tensor decomposition and the low-rank in non-local self-similarity direction underlying 4D similar clusters, we design a sparse tensor regularization term, which preserves the spatial-spectral structural correlation of HSIs. In addition, we present a proximal alternating direction method of multipliers (ADMM) based algorithm to efficiently solve the proposed model. Numerical experiments demonstrate that the proposed model outperforms many state-of-the-art HSI super-resolution methods.

Hyperspectral Image Superresolution Using Unidirectional Total Variation With Tucker Decomposition

The hyperspectral image superresolution (HSI-SR) problem aims to improve the spatial quality of a low spatial resolution HSI by fusing the LR-HSI with the corresponding high spatial resolution multispectral image. The generated HSI with high spatial quality, i.e., the target high spatial resolution hyperspectral image (HR-HSI), generally has some fundamental latent properties, e.g., the sparsity, and the piecewise smoothness along with the three modes (i.e., width, height, and spectral mode). However, limited works consider both properties in the HSI-SR problem. In this work, a novel unidirectional total variation (TV) based approach is been proposed. On the one hand, we consider that the target HR-HSI exhibits both the sparsity and the piecewise smoothness on the three modes, and they can be depicted well by the $\ell _1$ -norm and TV, respectively. On the other hand, we utilize the classical Tucker decomposition to decompose the target HR-HSI (a three-mode tensor) as a sparse core tensor multiplied by the dictionary matrices along with the three modes. Especially, we impose the $\ell _1$ -norm on core tensor to characterize the sparsity and the unidirectional TV on three dictionaries to characterize the piecewise smoothness. The proximal alternating optimization scheme and the alternating direction method of multipliers are used to iteratively solve the proposed model. Experiments on three common datasets illustrate that the proposed approach has better performance than some current state-of-the-art HSI-SR methods. Please find source code from: https://liangjiandeng.github.io/ .

A Truncated Matrix Decomposition for Hyperspectral Image Super-Resolution

Hyperspectral image super-resolution addresses the problem of fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) to produce a high-resolution hyperspectral image (HR-HSI). In this paper, we propose a novel fusion approach for hyperspectral image super-resolution by exploiting the specific properties of matrix decomposition, which consists of four main steps. First, an endmember extraction algorithm is used to extract an initial spectral matrix from LR-HSI. Then, with the initial spectral matrix, we estimate the spatial matrix, i.e., the spatial-contextual information, from the degraded observations of HR-HSI. Third, the spatial matrix is further utilized to estimate the spectral matrix from LR-HSI by solving a least squares (LS)-based problem. Finally, the target HR-HSI is constructed by combing the estimated spectral and spatial matrixes. In particular, two models are proposed to estimate the spatial matrix. One is a simple case that involves a LS-based problem, and the other is an elaborate case that consists of two fidelity terms and a spatial regularizer, where the spatial regularizer aiming to restrain the range of solutions is achieved by exploiting the superpixel-level low-rank characteristics of HR-HSI. Experiment results conducted on both synthetic and real data sets demonstrate the effectiveness of the proposed approach as compared to other hyperspectral image super-resolution methods.

Hyperspectral Image Super-Resolution Based on Tensor Spatial-Spectral Joint Correlation Regularization

Compared with natural image super-resolution, hyperspectral image super-resolution (HSR) is more complex because the redundancy in spectral bands and spatial information. To overcome the difficulties exist in HSR, in this paper, we propose a tensor spatial-spectral joint correlation based HSR method. Start with the tensor representation, we construct a series of fourth-order tensors to preserve the intrinsic structure of hyperspectral images, and then explore the spatial-spectral joint correlation based on meaningful interpretations of tensor canonical matrices. To further constrain the spectral characteristics, we analyze the sparsity of the spectral gradients and model it with Laplacian prior. Then, the two regularizations are combined with the reconstruction model to develop a new HSR method. Finally, an iterative optimization algorithm based on alternating direction method of multiplier (ADMM) and augmented Lagrangian multiplier method is proposed to reconstruct the high-resolution hyperspectral images. Experimental results on several data sets illustrate the effectiveness of our proposed method both in visual and numerical comparisons.

Simultaneous hyperspectral image super-resolution and geometric alignment with a hybrid camera system

Existing snapshot hyperspectral cameras usually suffer from limited spatial resolution in order to maintain reasonable temporal and spectral resolution. In contrast, the spatial resolution of commercial RGB cameras is quite high. Therefore, many methods have been developed for hyperspectral image (HSI) super-resolution by fusing a high resolution RGB image and a low resolution HSI. These methods have been extensively evaluated by using simulated image pair with exact geometric alignment. However, the effect of misalignment on these methods, which always arises in hybrid camera system, has rarely been investigated. In this paper, we present an effective approach for simultaneous HSI super-resolution and geometric alignment of the image pair with drastically contrasting spatial resolution. Besides, we also conduct a systematic evaluation of the misalignment effect on five state-of-the-art hybrid HSI super-resolution methods under five different geometric transformations and three benchmark datasets. Experimental results on both synthetic data and real images show that the proposed method outperforms the current state-of-the-art HSI super-resolution methods with a misaligned hybrid camera system in terms of both objective metric and subjective visual quality.

Spectral Representation vis Data-Guided Sparsity for Hyperspectral Image Super-Resolution†.

Hyperspectral imaging is capable of acquiring the rich spectral information of scenes and has great potential for understanding the characteristics of different materials in many applications ranging from remote sensing to medical imaging. However, due to hardware limitations, the existed hyper-/multi-spectral imaging devices usually cannot obtain high spatial resolution. This study aims to generate a high resolution hyperspectral image according to the available low resolution hyperspectral and high resolution RGB images. We propose a novel hyperspectral image superresolution method via non-negative sparse representation of reflectance spectra with a data guided sparsity constraint. The proposed method firstly learns the hyperspectral dictionary from the low resolution hyperspectral image and then transforms it into the RGB one with the camera response function, which is decided by the physical property of the RGB imaging camera. Given the RGB vector and the RGB dictionary, the sparse representation of each pixel in the high resolution image is calculated with the guidance of a sparsity map, which measures pixel material purity. The sparsity map is generated by analyzing the local content similarity of a focused pixel in the available high resolution RGB image and quantifying the spectral mixing degree motivated by the fact that the pixel spectrum of a pure material should have sparse representation of the spectral dictionary. Since the proposed method adaptively adjusts the sparsity in the spectral representation based on the local content of the available high resolution RGB image, it can produce more robust spectral representation for recovering the target high resolution hyperspectral image. Comprehensive experiments on two public hyperspectral datasets and three real remote sensing images validate that the proposed method achieves promising performances compared to the existing state-of-the-art methods.

Hyperspectral Image Super-Resolution via Adaptive Dictionary Learning and Double ℓ 1 Constraint

Hyperspectral image (HSI) super-resolution (SR) is an important technique for improving the spatial resolution of HSI. Recently, a method based on sparse representation improved the performance of HSI SR significantly. However, the spectral dictionary was learned under a fixed size, empirically, without considering the training data. Moreover, most of the existing methods fail to explore the relationship among the sparse coefficients. To address these crucial issues, an effective method for HSI SR is proposed in this paper. First, a spectral dictionary is learned, which can adaptively estimate a suitable size according to the input HSI without any prior information. Then, the proposed method exploits the nonlocal correlation of the sparse coefficients. Double ℓ 1 regularized sparse representation is then introduced to achieve better reconstructions for HSI SR. Finally, a high spatial resolution HSI is generated by the obtained coefficients matrix and the learned adaptive size spectral dictionary. To evaluate the performance of the proposed method, we conduct experiments on two famous datasets. The experimental results demonstrate that it can outperform some relatively state-of-the-art methods in terms of the popular universal quality evaluation indexes.

Hyperspectral image quality based on convolutional network of multi-scale depth

Hyperspectral imagery has been widely used in military and civilian research fields such as crop yield estimation, mineral exploration, and military target detection. However, for the limited imaging equipment and the complex imaging environment of hyperspectral images, the spatial resolution of hyperspectral images is still relatively low, which limits the application of hyperspectral images. So, studying the data characteristics of hyperspectral images deeply and improving the spatial resolution of hyperspectral images is an important prerequisite for accurate interpretation and wide application of hyperspectral images. The purpose of this paper is to deal with super-resolution of the hyperspectral image quickly and accurately, and maintain the spectral characteristics of the hyperspectral image, makes the spectral separability of the substrate in the original image remains unchanged after super-resolution processing. This paper first learns the mapping relationship between the spectral difference of low-resolution hyperspectral image and the spectral difference of the corresponding high-resolution hyperspectral image based on multiple scale convolutional neural network, Thus, apply this mapping relationship to the input low-resolution hyperspectral image generally, getting the corresponding high resolution spectral difference. Constrained space by using the image of reconstructed spectral difference, this requires the low-resolution hyperspectral image generated by the reconstructed image is to be close to the input low-resolution hyperspectral image in space, so that the whole process becomes a closed circulation system where the low-resolution hyperspectral image generation of high-resolution hyperspectral images, then back to low-resolution hyperspectral images. This innovative design further enhances the super-resolution performance of the algorithm. The experimental results show that the hyperspectral image super-resolution method based on convolutional neural network improves the input image spatial information, and the super-resolution performance of the model is above 90%, which can maintain the spectral information well.