Spatial-spectral Correlations Research Articles

Rethinking 3D-CNN in Hyperspectral Image Super-Resolution

Recently, CNN-based methods for hyperspectral image super-resolution (HSISR) have achieved outstanding performance. Due to the multi-band property of hyperspectral images, 3D convolutions are natural candidates for extracting spatial–spectral correlations. However, pure 3D CNN models are rare to see, since they are generally considered to be too complex, require large amounts of data to train, and run the risk of overfitting on relatively small-scale hyperspectral datasets. In this paper, we question this common notion and propose Full 3D U-Net (F3DUN), a full 3D CNN model combined with the U-Net architecture. By introducing skip connections, the model becomes deeper and utilizes multi-scale features. Extensive experiments show that F3DUN can achieve state-of-the-art performance on HSISR tasks, indicating the effectiveness of the full 3D CNN on HSISR tasks, thanks to the carefully designed architecture. To further explore the properties of the full 3D CNN model, we develop a 3D/2D mixed model, a popular kind of model prior, called Mixed U-Net (MUN) which shares a similar architecture with F3DUN. Through analysis on F3DUN and MUN, we find that 3D convolutions give the model a larger capacity; that is, the full 3D CNN model can obtain better results than the 3D/2D mixed model with the same number of parameters when it is sufficiently trained. Moreover, experimental results show that the full 3D CNN model could achieve competitive results with the 3D/2D mixed model on a small-scale dataset, suggesting that 3D CNN is less sensitive to data scaling than what people used to believe. Extensive experiments on two benchmark datasets, CAVE and Harvard, demonstrate that our proposed F3DUN exceeds state-of-the-art HSISR methods both quantitatively and qualitatively.

Feedback Refined Local-Global Network for Super-Resolution of Hyperspectral Imagery

Powered by advanced deep-learning technology, multi-spectral image super-resolution methods based on convolutional neural networks have recently achieved great progress. However, the single hyperspectral image super-resolution remains a challenging problem due to the high-dimensional and complex spectral characteristics of hyperspectral data, which make it difficult for general 2D convolutional neural networks to simultaneously capture spatial and spectral prior information. To deal with this issue, we propose a novel Feedback Refined Local-Global Network (FRLGN) for the super-resolution of hyperspectral image. To be specific, we develop a new Feedback Structure and a Local-Global Spectral block to alleviate the difficulty in spatial and spectral feature extraction. The Feedback Structure can transfer the high-level information to guide the generation process of low-level features, which is achieved by a recurrent structure with finite unfoldings. Furthermore, in order to effectively use the high-level information passed back, a Local-Global Spectral block is constructed to handle the feedback connections. The Local-Global Spectral block utilizes the feedback high-level information to correct the low-level feature from local spectral bands and generates powerful high-level representations among global spectral bands. By incorporating the Feedback Structure and Local-Global Spectral block, the FRLGN can fully exploit spatial-spectral correlations among spectral bands and gradually reconstruct high-resolution hyperspectral images. Experimental results indicate that FRLGN presents advantages on three public hyperspectral datasets.

Fast reconstruction of hyperspectral images from coded acquisitions using a separability assumption

We present a fast reconstruction algorithm for hyperspectral images, utilizing a small amount of data without the need for any training. The method is implemented with a dual disperser hyperspectral imager and makes use of spatial-spectral correlations by a so-called separability assumption that assumes that the image is made of regions of homogenous spectra. The reconstruction algorithm is simple and ready-to-use and does not require any prior knowledge of the scene. A simple proof-of-principle experiment is performed, demonstrating that only a small number of acquisitions are required, and the resulting compressed data-cube is reconstructed near instantaneously.

Spatial-spectral correlations of broadband speckle in around-the-corner imaging conditions.

Correlations of broadband speckle have important implications for passive, non-line-of-sight imaging. We examine the spectral and spatial correlations of broadband, around-the-corner speckle and reveal a set of equations that locate the spatial maximum of the paraxial spatial-spectral correlation function. We confirm the validity of the spatial-spectral correlation framework through experiment, theory and simulation.

Hyperspectral Image Super-Resolution via Recurrent Feedback Embedding and Spatial–Spectral Consistency Regularization

Hyperspectral images with tens to hundreds of spectral bands usually suffer from low spatial resolution due to the limitation of the amount of incident energy. Without auxiliary images, the single hyperspectral image super-resolution (SR) method is still a challenging problem because of the high-dimensionality characteristic and special spectral patterns of hyperspectral images. Failing to thoroughly explore the coherence among hyperspectral bands and preserve the spatial–spectral structure of the scene, the performance of existing methods is still limited. In this article, we propose a novel single hyperspectral image SR method termed RFSR, which models the spectrum correlations from a sequence perspective. Specifically, we introduce a recurrent feedback network to fully exploit the complementary and consecutive information among the spectra of the hyperspectral data. With the group strategy, each grouping band is first super-resolved by exploring the consecutive information among groups via feedback embedding. For better preservation of the spatial–spectral structure among hyperspectral data, a regularization network is subsequently appended to enforce spatial–spectral correlations over the intermediate estimation. Experimental results on both natural and remote sensing hyperspectral images demonstrate the advantage of our approach over the state-of-the-art methods.

Hyperspectral Snapshot Compressive Imaging with Non-Local Spatial-Spectral Residual Network

Snapshot Compressive Imaging is an emerging technology that is based on compressive sensing theory to achieve high-efficiency hyperspectral data acquisition. The core problem of this technology is how to reconstruct 3D hyperspectral data from the 2D snapshot measurement in a fast and high-quality manner. In this paper, we propose a novel deep network, which consists of the symmetric residual module and the non-local spatial-spectral attention module, to learn the reconstruction mapping in a data-driven way. The symmetric residual module uses symmetric residual connections to improve the potential of interaction between convolution operations and further promotes the fusion of local features. The non-local spatial-spectral attention module is designed to capture the non-local spatial-spectral correlation in the hyperspectral image. Specifically, this module calculates the channel attention matrix to capture the global correlations between all of the spectral channels, and it fuses the channel attention attained feature maps and the spatial attention weighted features as the module output, thus both of the spatial-spectral correlations of hyperspectral images can be fully utilized for reconstruction. In addition, a compound loss, including the reconstruction loss, the measurement loss, and the cosine loss, is designed to guide the end-to-end network learning. We experimentally evaluate the proposed method on simulation and real datasets. The experimental results show that the proposed network outperforms the competing methods in terms of the reconstruction quality and running time.

Key-Frame Detection and Super-Resolution of Hyperspectral Video via Sparse-Based Cumulative Tensor Factorization

Thanks to the rapid development of hyperspectral sensors, hyperspectral videos (HSV) can now be collected with high temporal and spectral resolutions and utilized to handle invisible dynamic monitoring missions, such as chemical gas plume tracking. However, using such sequential large-scale data effectively is challenged, because the direct process of these data requires huge demands in terms of computational loads and memory. This paper presents a key-frame and target-detecting algorithm based on cumulative tensor CANDECOMP/PARAFAC (CP) factorization (CTCF) to select the frames where the target shows up, and a novel super-resolution (SR) method using sparse-based tensor Tucker factorization (STTF) is used to improve the spatial resolution. In the CTCF method, the HSV sequence is seen as cumulative tensors and the correlation of adjacent frames is exploited by applying CP tensor approximation. In the proposed STTF-based SR method, we consider the HSV frame as a third-order tensor; then, HSV frame super-resolution problem is transformed into estimations of the dictionaries along three dimensions and estimation of the core tensor. In order to promote sparse core tensors, a regularizer is incorporated to model the high spatial-spectral correlations. The estimations of the core tensor and the dictionaries along three dimensions are formulated as sparse-based Tucker factorizations of each HSV frame. Experimental results on real HSV data set demonstrate the superiority of the proposed CTCF and STTF algorithms over the comparative state-of-the-art target detection and SR approaches.

Fast stripe noise removal from hyperspectral image via multi-scale dilated unidirectional convolution

Hyperspectral images (HSIs) are often contaminated by noises due to the multi-detector imaging systems, which greatly affects the subsequent HSIs interpretation and application. The 3D HSIs deliver extra spectral information, which makes the most existing destriping algorithms hardly satisfied, and the complete stripes removal and less test time consuming remain to be overcome. To meet these challenges, we present a multi-scale dilated unidirectional convolution network (MsDUC) with the following contributions. First, the deep learning-based method can fully exploit and preserve spatial-spectral correlations in 3D HSIs while the conventional methods failed to realize it. Second, different dilated convolution learns different scale features, so the introduced multi-scale dilated convolution could get more contextual information for the final restoration. Third, the clear directional signature of stripe noise and the unidirectional total variation (UTV) model inspired us to put forward the unidirectional convolution to capture the directional signature of stripe, meanwhile, the less trainable parameters and the utilized residual strategy speed up the learning process. Experimental results have shown that our method outperforms many of the state-of-the-art methods in both image restoration performance and test running time. Our code can be download from .

Cloud and Cloud Shadow Detection Based on Multiscale 3D-CNN for High Resolution Multispectral Imagery

Cloud and cloud shadow detection is one of the most important tasks for optical remote sensing image preprocessing. It is not an easy task due to the variety and complexity of underlying surfaces, such as the low-albedo objects (water and mountain shadow) and the high-albedo objects (snow and ice). In this study, an end-to-end multiscale 3D-CNN method is proposed for cloud and cloud shadow detection in high resolution multispectral imagery. Specifically, a multiscale learning module is designed to extract cloud and cloud shadow contextual information of different levels. In order to make full use of band information, four band-combination images are inputted into the multiscale 3D-CNN. A joint spectral-spatial information of 3D-convolution layer is developed to fully explore the joint spatial-spectral correlations feature in the input data. Overall, in the experiments undertaken in this paper, the proposed method achieved a mean overall accuracy of 97.27% for cloud detection, with a mean precision of 96.02% and a mean recall of 95.86%. For cloud shadow detection, the proposed method achieved a mean precision of 95.92% and a mean recall of 92.86%. Experimental results on two validation datasets (GF-1 WFV validation data and ZY-3 validation data) show that the proposed multiscale-3D-CNN method achieved good performance with limited spectral ranges.

Deep Spatial–Spectral Representation Learning for Hyperspectral Image Denoising

Deep learning has found successful applications in restoration of two-dimensional (2-D) images including denoising, dehazing, and superresolution. However, existing deep convolutional neural network (DCNN) architecture cannot fully exploit spatial–spectral correlations in three-dimensional (3-D) hyperspectral images (HSIs) (directly extending 2-D DCNN into 3-D will significantly increase computational complexity); meantime, unlike 2-D images, there is an obstacle caused by the shortage of training data for HSIs. To meet those challenges, we present a novel, deep-learning framework for 3-D HSI denoising with the following contributions. First, inspired by the success of U-net in low-dose current-transformer denoising, we propose a novel approach of encoding rich multi-scale information of HSIs by a modified 3-D U-net. Second, we present a computationally efficient implementation of 3-D U-net based on the strategy of separable filtering. By decomposing 3-D filtering into 2-D spatial filtering and 1-D spectral filtering, we can achieve substantial savings on the number of network parameters to keep the computational complexity low. Third, we have developed a transfer learning approach of synthetically generating HSIs from RGB images as supplementary training data. The synthesized HSIs are used for the initial training of the modified 3-D U-net denoising network, which will be fine-tuned on real HSI images. Experimental results have shown that the proposed 3-D U-net denoising method significantly outperforms existing model-based HSI denoising methods.

Distributions of Silica and Biopolymer Structural Components in the Spore Elater of Equisetum arvense, an Ancient Silicifying Plant.

Equisetum species are primitive vascular plants that benefit from the biogenesis of silica bio-organic inclusions in their tissues and participate in the annual biosilica turnover in local eco-systems. As means of Equisetum reproduction and propagation, spores are expected to reflect the evolutionary adaptation of the plants to the climatic conditions at different times of the year. Combining methods of Raman and scanning electron microscopy and assisted with density functional theory, we conducted material spatial-spectral correlations to characterize the distribution of biopolymers and silica based structural elements that contribute to the bio-mineral content of the elater. The elater tip has underlying skeletal-like structural elements where cellulose fibers provide strength and flexibility, both of which are necessary for locomotion. The surface of the elater tips is rich with less ordered pectin like polysaccharide and shows a ridged, folded character. At the surface we observe silica of amorphous, colloidal form in nearly spherical structures where the silica is only a few layers thick. We propose the observed expansion of elater tips upon germination and the form of silica including encapsulated biopolymers are designed for ready dispersion, release of the polysaccharide-arginine rich content and to facilitate silica uptake to the developing plant. This behavior would help to condition local soil chemistry to facilitate competitive rooting potential and stem propagation.

Fusing Hyperspectral and Multispectral Images via Coupled Sparse Tensor Factorization.

Fusing a low spatial resolution hyperspectral image (LR-HSI) with a high spatial resolution multispectral image (HR-MSI) to obtain a high spatial resolution hyperspectral image (HR-HSI) has attracted increasing interest in recent years. In this paper, we propose a coupled sparse tensor factorization (CSTF) based approach for fusing such images. In the proposed CSTF method, we consider an HR-HSI as a three-dimensional tensor and redefine the fusion problem as the estimation of a core tensor and dictionaries of the three modes. The high spatial-spectral correlations in the HR-HSI are modeled by incorporating a regularizer which promotes sparse core tensors. The estimation of the dictionaries and the core tensor are formulated as a coupled tensor factorization of the LR-HSI and of the HR-MSI. Experiments on two remotely sensed HSIs demonstrate the superiority of the proposed CSTF algorithm over current state-of-the-art HSI-MSI fusion approaches.

Demosaicing Based on Directional Difference Regression and Efficient Regression Priors.

Color demosaicing is a key image processing step aiming to reconstruct the missing pixels from a recorded raw image. On the one hand, numerous interpolation methods focusing on spatial-spectral correlations have been proved very efficient, whereas they yield a poor image quality and strong visible artifacts. On the other hand, optimization strategies, such as learned simultaneous sparse coding and sparsity and adaptive principal component analysis-based algorithms, were shown to greatly improve image quality compared with that delivered by interpolation methods, but unfortunately are computationally heavy. In this paper, we propose efficient regression priors as a novel, fast post-processing algorithm that learns the regression priors offline from training data. We also propose an independent efficient demosaicing algorithm based on directional difference regression, and introduce its enhanced version based on fused regression. We achieve an image quality comparable to that of the state-of-the-art methods for three benchmarks, while being order(s) of magnitude faster.

Joint Color Decrosstalk and Demosaicking for CFA Cameras

In interest of low cost, low power consumption, and compact size, most digital cameras adopt a design of single sensor array coupled with a color filter array. This design inevitably suffers, due to physical characteristics of the optical and semiconductor components and the imperfection of manufacturing, from the problem of crosstalk between different color channels. Channel crosstalk can desaturate colors and blur image details, but the problem was seemingly overlooked by existing color demosaicking algorithms. To rectify this deficiency we propose a new joint demosaicking and decrosstalk technique that counters the effects of channel crosstalks by adaptive least-squares inverse filtering. The new technique integrates the operations of deconvolution for crosstalk removal and interpolation for color demosaicking, and it introduces a general framework in which any spatially varying crosstalks and varying spatial-spectral correlations can be modeled and factored into the color reproduction. Simulation results show that the proposed technique is highly effective and capable to obtain both high color fidelity and sharp, clean spatial details.