High Resolution Hyperspectral Image Research Articles

Multispectral and Hyperspectral Image Fusion Based on Regularized Coupled Non-Negative Block-Term Tensor Decomposition

The problem of multispectral and hyperspectral image fusion (MHF) is to reconstruct images by fusing the spatial information of multispectral images and the spectral information of hyperspectral images. Focusing on the problem that the hyperspectral canonical polyadic decomposition model and the Tucker model cannot introduce the physical interpretation of the latent factors into the framework, it is difficult to use the known properties and abundance of endmembers to generate high-quality fusion images. This paper proposes a new fusion algorithm. In this paper, a coupled non-negative block-term tensor model is used to estimate the ideal high spatial resolution hyperspectral images, its sparsity is characterized by adding 1-norm, and total variation (TV) is introduced to describe piecewise smoothness. Secondly, the different operators in two directions are defined and introduced to characterize their piecewise smoothness. Finally, the proximal alternating optimization (PAO) algorithm and the alternating multiplier method (ADMM) are used to iteratively solve the model. Experiments on two standard datasets and two local datasets show that the performance of this method is better than the state-of-the-art methods.

Detecting Xylella fastidiosa in a machine learning framework using Vcmax and leaf biochemistry quantified with airborne hyperspectral imagery

The bacterium Xylella fastidiosa (Xf) is a plant pathogen that can block the flow of water and nutrients through the xylem. Xf symptoms may be confounded with generic water stress responses. Here, we assessed changes in biochemical, biophysical and photosynthetic traits, inferred using biophysical models, in Xf-affected almond orchards under rainfed and irrigated conditions on the Island of Majorca (Balearic Islands, Spain). Recent research has demonstrated the early detection of Xf-infections by monitoring spectral changes associated with pigments, canopy structural traits, fluorescence emission and transpiration. Nevertheless, there is still a need to make further progress in monitoring physiological processes (e.g., photosynthesis rate) to be able to efficiently detect when Xf-infection causes subtle spectral changes in photosynthesis. This paper explores the ability of parsimonious machine learning (ML) algorithms to detect Xf-infected trees operationally, when considering a proxy of photosynthetic capacity, namely the maximum carboxylation rate (Vcmax), along with carbon-based constituents (CBC, including lignin), and leaf biochemical traits and tree-crown temperature (Tc) as an indicator of transpiration rates. The ML framework proposed here reduced the uncertainties associated with the extraction of reflectance spectra and temperature from individual tree crowns using high-resolution hyperspectral and thermal images. We showed that the relative importance of Vcmax and leaf biochemical constituents (e.g., CBC) in the ML model for the detection of Xf at early stages of development were intrinsically associated with the water and nutritional conditions of almond trees. Overall, the functional traits that were most consistently altered by Xf-infection were Vcmax, pigments, CBC, and Tc, and, particularly in rainfed-trees, anthocyanins, and Tc. The parsimonious ML model for Xf detection yielded accuracies exceeding 90% (kappa = 0.80). This study brings progress in the development of an operational ML framework for the detection of Xf outbreaks based on plant traits related to photosynthetic capacity, plant biochemistry and structural decay parameters.

Hyperspectral image super-resolution via a multi-stage scheme without employing spatial degradation.

Recently, it has become popular to obtain a high spatial resolution hyperspectral image (HR-HSI) by fusing a low spatial resolution hyperspectral image (LR-HSI) with a high spatial resolution RGB image (HR-RGB). Existing HSI super-resolution methods are designed based on a known spatial degeneration. In practice, it is difficult to obtain correct spatial degradation, which restricts the performance of existing methods. Therefore, we propose a multi-stage scheme without employing the spatial degradation model. The multi-stage scheme consists of three stages: initialization, modification, and refinement. According to the angle similarity between the HR-RGB pixel and LR-HSI spectra, we first initialize a spectrum for each HR-RGB pixel. Then, we propose a polynomial function to modify the initialized spectrum so that the RGB color values of the modified spectrum are the same as the HR-RGB. Finally, the modified HR-HSI is refined by a proposed optimization model, in which a novel, to the best of our knowledge, spectral-spatial total variation (SSTV) regularizer is investigated to keep the spectral and spatial structure of the reconstructed HR-HSI. The experimental results on two public datasets and our real-world images demonstrate our method outperforms eight state-of-the-art existing methods in terms of both reconstruction accuracy and computational efficiency.

Coupled Tensor Block Term Decomposition with Superpixel-Based Graph Laplacian Regularization for Hyperspectral Super-Resolution

Hyperspectral image (HSI) super-resolution aims at improving the spatial resolution of HSI by fusing a high spatial resolution multispectral image (MSI). To preserve local submanifold structures in HSI super-resolution, a novel superpixel graph-based super-resolution method is proposed. Firstly, the MSI is segmented into superpixel blocks to form two-directional feature tensors, then two graphs are created using spectral–spatial distance between the unfolded feature tensors. Secondly, two graph Laplacian terms involving underlying BTD factors of high-resolution HSI are developed, which ensures the inheritance of the spatial geometric structures. Finally, by incorporating graph Laplacian priors with the coupled BTD degradation model, a HSI super-resolution model is established. Experimental results demonstrate that the proposed method achieves better fused results compared with other advanced super-resolution methods, especially on the improvement of the spatial structure.

基于高光谱荒漠草原的微斑块光谱特征分析与提取

In hyperspectral remote sensing images, desert steppe vegetation, bare soil, and rat holes appear as micro-patches. The spectral feature analysis of micro-patches is the basis for identification and classification and also the basis for quantitative remote sensing monitoring of ground objects. Inner Mongolia desert steppe micro-patch as the research object extracts the spectral reflectance of different micro-patches, performs various vegetation index calculations, quantitatively analyzes the spectral characteristics of different micro-patches, and proposes a micro-patch spectral analysis method. Classification of high-resolution hyperspectral images of desert steppe surface micropatches. The results show that: (1) There are pronounced differences in the spectral reflectance of the three types of surface micro-patches. The vegetation has apparent characteristics in the green wave reflection peak and the red wave absorption valley. The spectral reflectance of the bare soil is higher than that of the mouse hole, and the two have been increasing. The trend is increasing slowly; (2) The proposal and application of the MSA index can effectively realize the identification and classification of surface micropatches, and the Kappa coefficient has reached 0.906 through confusion matrix verification. The above spectral analysis method realizes the classification and identification of complex ground objects using near-ground remote sensing images. It provides new ideas and methods for accurate quantitative statistics of desert grassland ecological information.

Deep Pansharpening via 3D Spectral Super-Resolution Network and Discrepancy-Based Gradient Transfer

High-resolution (HR) multispectral (MS) images contain sharper detail and structure compared to the ground truth high-resolution hyperspectral (HS) images. In this paper, we propose a novel supervised learning method, which considers pansharpening as the spectral super-resolution of high-resolution multispectral images and generates high-resolution hyperspectral images. The proposed method learns the spectral mapping between high-resolution multispectral images and the ground truth high-resolution hyperspectral images. To consider the spectral correlation between bands, we build a three-dimensional (3D) convolution neural network (CNN). The network consists of three parts using an encoder–decoder framework: spatial/spectral feature extraction from high-resolution multispectral images/low-resolution (LR) hyperspectral images, feature transform, and image reconstruction to generate the results. In the image reconstruction network, we design the spatial–spectral fusion (SSF) blocks to reuse the extracted spatial and spectral features in the reconstructed feature layer. Then, we develop the discrepancy-based deep hybrid gradient (DDHG) losses with the spatial–spectral gradient (SSG) loss and deep gradient transfer (DGT) loss. The spatial–spectral gradient loss and deep gradient transfer loss are developed to preserve the spatial and spectral gradients from the ground truth high-resolution hyperspectral images and high-resolution multispectral images. To overcome the spectral and spatial discrepancy between two images, we design a spectral downsampling (SD) network and a gradient consistency estimation (GCE) network for hybrid gradient losses. In the experiments, it is seen that the proposed method outperforms the state-of-the-art methods in the subjective and objective experiments in terms of the structure and spectral preservation of high-resolution hyperspectral images.

Multi-Resolution Collaborative Fusion of SAR, Multispectral and Hyperspectral Images for Coastal Wetlands Mapping

The hyperspectral, multispectral, and synthetic aperture radar (SAR) remote sensing images provide complementary advantages in high spectral resolution, high spatial resolution, and geometric and polarimetric properties, generally. How to effectively integrate cross-modal information to obtain a high spatial resolution hyperspectral image with the characteristics of the SAR is promising. However, due to divergent imaging mechanisms of modalities, existing SAR and optical image fusion techniques generally remain limited due to the spectral or spatial distortions, especially for complex surface features such as coastal wetlands. This paper provides, for the first time, an efficient multi-resolution collaborative fusion method for multispectral, hyperspectral, and SAR images. We improve generic multi-resolution analysis with spectral-spatial weighted modulation and spectral compensation to achieve minimal spectral loss. The backscattering gradients of SAR are guided to fuse, which is calculated from saliency gradients with edge preserving. The experiments were performed on ZiYuan-1 02D (ZY-1 02D) and GaoFen-5B (AHSI) hyperspectral, Sentinel-2 and GaoFen-5B (VIMI) multispectral, and Sentinel-1 SAR images in the challenging coastal wetlands. Specifically, the fusion results were comprehensively tested and verified on the qualitative, quantitative, and classification metrics. The experimental results show the competitive performance of the proposed method.

The effects of spectral dimensionality reduction on hyperspectral pixel classification: A case study.

This paper presents a systematic study of the effects of hyperspectral pixel dimensionality reduction on the pixel classification task. We use five dimensionality reduction methods-PCA, KPCA, ICA, AE, and DAE-to compress 301-dimensional hyperspectral pixels. Compressed pixels are subsequently used to perform pixel classifications. Pixel classification accuracies together with compression method, compression rates, and reconstruction errors provide a new lens to study the suitability of a compression method for the task of pixel classification. We use three high-resolution hyperspectral image datasets, representing three common landscape types (i.e. urban, transitional suburban, and forests) collected by the Remote Sensing and Spatial Ecosystem Modeling laboratory of the University of Toronto. We found that PCA, KPCA, and ICA post greater signal reconstruction capability; however, when compression rates are more than 90% these methods show lower classification scores. AE and DAE methods post better classification accuracy at 95% compression rate, however their performance drops as compression rate approaches 97%. Our results suggest that both the compression method and the compression rate are important considerations when designing a hyperspectral pixel classification pipeline.

Hyperspectral image super-resolution based on the transfer of both spectra and multi-level features.

Existing hyperspectral image (HSI) super-resolution methods fusing a high-resolution RGB image (HR-RGB) and a low-resolution HSI (LR-HSI) always rely on spatial degradation and handcrafted priors, which hinders their practicality. To address these problems, we propose a novel, to the best of our knowledge, method with two transfer models: a window-based linear mixing (W-LM) model and a feature transfer model. Specifically, W-LM initializes a high-resolution HSI (HR-HSI) by transferring the spectra from the LR-HSI to the HR-RGB. By using the proposed feature transfer model, the HR-RGB multi-level features extracted by a pre-trained convolutional neural network (CNN) are then transferred to the initialized HR-HSI. The proposed method fully exploits spectra of LR-HSI and multi-level features of HR-RGB and achieves super-resolution without requiring the spatial degradation model and any handcrafted priors. The experimental results for 32 × super-resolution on two public datasets and our real image set demonstrate the proposed method outperforms eight state-of-the-art existing methods.

High-Resolution Hyperspectral Imaging Using Low-Cost Components: Application within Environmental Monitoring Scenarios.

High-resolution hyperspectral imaging is becoming indispensable, enabling the precise detection of spectral variations across complex, spatially intricate targets. However, despite these significant benefits, currently available high-resolution set-ups are typically prohibitively expensive, significantly limiting their user base and accessibility. These limitations can have wider implications, limiting data collection opportunities, and therefore our knowledge, across a wide range of environments. In this article we introduce a low-cost alternative to the currently available instrumentation. This instrument provides hyperspectral datasets capable of resolving spectral variations in mm-scale targets, that cannot typically be resolved with many existing low-cost hyperspectral imaging alternatives. Instrument metrology is provided, and its efficacy is demonstrated within a mineralogy-based environmental monitoring application highlighting it as a valuable addition to the field of low-cost hyperspectral imaging.

Semi-blind hyperspectral and multispectral image fusion based on a non-factorization model

Fusion of low-resolution hyperspectral images (LR-HSI) with high-resolution multispectral images (HR-MSI) is currently the most cost-effective method for acquiring high-resolution hyperspectral images (HR-HSI). The existing fusion models like spectral unmixing, matrix-factorization, and tensor-factorization require decomposing the target HR-HSI into multiple factor matrices or tensors before fusion. Usually requires a large number of variable optimization processes. So, the above approaches do not apply to the arguably real-time scenario. Moreover, the miscellaneous intermediate estimates render poor fusion stability. This paper proposes a novel non-factorization model to address the above problems. The model firstly gets access to a rough estimate of HR-HSI through the Moore–Penrose inverse operation. Then, we acquire the further HR-HSI forecast according to the correlation between the target HR-HSI and LR-HSI singular values through the coupled singular value decomposition (SVD) of the rough HR-HSI and LR-HIS. Finally, we obtain the precise HR-HSI estimates via the multiplicative update rule. Our proposed model directly estimates the target HR-HSI, reduces the estimation process of intermediate variables, effectively saves calculation time, and enhances the stability of the fusion. We conduct experiments on remote sensing, ground-based, and real datasets to validate our proposed model’s real-time and stability.

Heterogeneity-Aware Recurrent Neural Network for Hyperspectral and Multispectral Image Fusion

Due to the hardware limitations of remote imaging sensors, it is challenging to acquire images with high resolution in both the spatial and spectral domains. An effective and economical way to obtain high-resolution hyperspectral images (HR HSI) is to fuse low-resolution hyperspectral images (LR HSI) and high-resolution multispectral images (HR MSI). However, most existing deep learning based fusion methods employ the same network for all of the spectra without exploring their complex regional heterogeneity of hyperspectral characteristics. Taking various intrinsic spatial and spectral characteristics across different pixels into consideration, this paper proposes a mixture of recurrent neural networks (RNNs) under the variational probabilistic framework for spatial and spectral resolution enhancement. More specifically, we firstly cluster spectral characteristics into different groups, and employ different RNN experts for various spectra generation under the guidance of clustering. Moreover, a cluster-specific learnable Gaussian prior is proposed to provide a prior knowledge of heterogeneity. Further, an online variational inference scheme is derived for end-to-end optimization. Extensive experimental results demonstrate the effectiveness and efficiency of the proposed model on both synthetic and real datasets, compared with state-of-the-art unsupervised fusion methods.

Unsupervised super-resolution reconstruction of hyperspectral histology images for whole-slide imaging.

.SignificanceHyperspectral imaging (HSI) provides rich spectral information for improved histopathological cancer detection. However, acquiring high-resolution HSI data for whole-slide imaging (WSI) can be time-consuming and requires a huge amount of storage space.AimWSI using a color camera can be achieved with fast speed, high image resolution, and excellent image quality due to the established techniques. We aim to develop an RGB-guided unsupervised hyperspectral super-resolution reconstruction method that is hypothesized to improve image quality while maintaining the spectral characteristics.ApproachHigh-resolution hyperspectral images of 32 histologic slides were obtained via automated WSI. High-resolution RGB histology images were registered to the hyperspectral images for RGB guidance. An unsupervised super-resolution network was trained to take the downsampled low-resolution hyperspectral patches (LR-HSI) and high-resolution RGB patches (HR-RGB) as inputs to reconstruct high-resolution hyperspectral patches (HR-HSI). Then, an Inception-based network was trained with the HR-RGB, original HR-HSI, and generated HR-HSI, respectively, for whole-slide histopathological cancer detection.ResultsOur super-resolution reconstruction network generated high-resolution hyperspectral images with well-maintained spectral characteristics and improved image quality. Image classification using the original hyperspectral data outperformed RGB because of the extra spectral information. The generated hyperspectral image patches further improved the results.ConclusionsThe proposed method potentially reduces image acquisition time, saves storage space without compromising image quality, and improves the image classification performance.

Estimating Chemical Oxygen Demand in estuarine urban rivers using unmanned aerial vehicle hyperspectral images

In this study, we combined ground-based hyperspectral data, unmanned aerial vehicles (UAVs) remotely sensed hyperspectral images, and 1D-CNN algorithms to quantitatively characterize and estimate the Chemical Oxygen Demand (COD) of estuarine urban rivers. The spectral response mechanism of COD is imprecise due to its complex composition; however, we found that hyperspectral remote sensing data could be used for COD monitoring because of the data's rich spectral information. The potential of hyperspectral sensors installed on UAVs to estimate and map the COD of urban rivers has not been thoroughly explored. We used in situ above-water hyperspectral data from 498 sites and synchronous water samples in band ratio, SVM, and 1D-CNN algorithms to build retrieval models. We found that the 1D-CNN model performed the best with an R2 of 0.78 and an RMSE of 5.22 when using the original reflectance data as input. The 1D-CNN model may also have a better ability to identify water samples with abnormally high concentrations. Our results revealed that transferring the ground-based derived 1D-CNN retrieval model for COD to the high-resolution hyperspectral images is a reliable method for determining COD from the images. We concluded that UAV remotely sensed hyperspectral images are valuable for COD concentration monitoring and mapping, critical to urban water quality management decision-making.

Feedback Refined Local-Global Network for Super-Resolution of Hyperspectral Imagery

Powered by advanced deep-learning technology, multi-spectral image super-resolution methods based on convolutional neural networks have recently achieved great progress. However, the single hyperspectral image super-resolution remains a challenging problem due to the high-dimensional and complex spectral characteristics of hyperspectral data, which make it difficult for general 2D convolutional neural networks to simultaneously capture spatial and spectral prior information. To deal with this issue, we propose a novel Feedback Refined Local-Global Network (FRLGN) for the super-resolution of hyperspectral image. To be specific, we develop a new Feedback Structure and a Local-Global Spectral block to alleviate the difficulty in spatial and spectral feature extraction. The Feedback Structure can transfer the high-level information to guide the generation process of low-level features, which is achieved by a recurrent structure with finite unfoldings. Furthermore, in order to effectively use the high-level information passed back, a Local-Global Spectral block is constructed to handle the feedback connections. The Local-Global Spectral block utilizes the feedback high-level information to correct the low-level feature from local spectral bands and generates powerful high-level representations among global spectral bands. By incorporating the Feedback Structure and Local-Global Spectral block, the FRLGN can fully exploit spatial-spectral correlations among spectral bands and gradually reconstruct high-resolution hyperspectral images. Experimental results indicate that FRLGN presents advantages on three public hyperspectral datasets.

Hyperspectral retrievals of suspended sediment using cluster-based machine learning regression in shallow waters

Remote sensing of suspended sediment in shallow waters is challenging because of the increased optical variability of the water, resulting from the influence of suspended matter in the water column and the heterogeneous bottom properties. To overcome this limitation, in this study, we developed a novel framework called cluster-based machine learning regression for optical variability (CMR-OV), using the Gaussian mixture model (GMM) clustering technique and a random forest regressor (RFR). We evaluated the model using an optically complex dataset from a field-scale experiment. This experiment was conducted with four sediment types injected into an experimental meandering channel divided into two reaches with submerged vegetation and a natural sand bottom. We obtained high-resolution hyperspectral images using unmanned aerial vehicles (UAVs) and measured the in situ suspended sediment concentration using laser diffraction sensors. Based on optical similarity, we used CMR-OV to divide the hyperspectral dataset into several clusters. Then, we built separate RFR models for each cluster using the corresponding spectral bands that were selected using recursive feature elimination (RFE). Thus, we found that the proposed CMR-OV yielded superior results compared to the conventional RFR model, decreasing the total error score by 10.81%. The optical spectral bands of each cluster were distinguished from each other, indicating that the datasets that were spectrally discriminated from clustering enhanced the performance of the estimator. By comparing the clustered spectral dataset and physical factors, we proved the bottom type was the most critical factor in separating the clusters, even though the variability in the sediment properties also induced substantial spectral changes. Our findings demonstrated that CMR-OV accurately reproduced the spatiotemporal distribution of suspended sediment under optically complex conditions by addressing the heterogeneity of bottom reflectance in shallow water.

Correction of Dropped Frames in High-resolution Push-broom Hyperspectral Images for Cultural Heritage

Dropped frames can occur in line-scan cameras, which result in non-uniform spatial sampling of the scene. A dropped frame occurs when data from an image sensor is not successfully recorded. When mosaicking multiple line-scan images, such as in high-resolution imaging, this can cause misalignment. Much previous work to identify dropped frames in video prioritises fast computation over high accuracy, whereas in heritage imaging, high accuracy is often preferred over short computation time. Two approaches to identify the position of dropped frames are presented, both using the A* search algorithm to correct dropped frames. One method aligns overlapping sections of push-broom images and the other aligns the push-broom image to a lower resolution reference image. The two methods are compared across a range of test images, and the method aligning overlapping sections is shown to perform better than the method using a reference image under most circumstances. The overlap method was applied to hyperspectral images acquired of La Ghirlandata, an 1873 oil on canvas painting by D. G. Rossetti, enabling a high-resolution hyperspectral image mosaic to be produced. The resulting composite image is 10,875 \( \times \) 14,697 pixels each with 500 spectral bands from 400–2,500 nm. This corresponds to a spatial resolution of \( 80 \,\mathrm{\upmu }\mathrm{m} \) and a spectral resolution of 3–6 nm.

Unsupervised Super Resolution Network for Hyperspectral Histologic Imaging.

Hyperspectral imaging (HSI) has many advantages in microscopic applications, including high sensitivity and specificity for cancer detection on histological slides. However, acquiring hyperspectral images of a whole slide with a high image resolution and a high image quality can take a long scanning time and require a very large data storage. One potential solution is to acquire and save low-resolution hyperspectral images and reconstruct the high-resolution ones only when needed. The purpose of this study is to develop a simple yet effective unsupervised super resolution network for hyperspectral histologic imaging with the guidance of RGB digital histology images. High-resolution hyperspectral images of hemoxylin & eosin (H&E) stained slides were obtained at 10× magnification and down-sampled 2×, 4×, and 5× to generate low-resolution hyperspectral data. High-resolution digital histologic RGB images of the same field of view (FOV) were cropped and registered to the corresponding high-resolution hyperspectral images. A neural network based on a modified U-Net architecture, which takes the low-resolution hyperspectral images and high-resolution RGB images as inputs, was trained with unsupervised methods to output high-resolution hyperspectral data. The generated high-resolution hyperspectral images have similar spectral signatures and improved image contrast than the original high-resolution hyperspectral images, which indicates that the super resolution network with RGB guidance can improve the image quality. The proposed method can reduce the acquisition time and save storage space taken up by hyperspectral images without compromising image quality, which will potentially promote the use of hyperspectral imaging technology in digital pathology and many other clinical applications.

A model-guided deep convolutional sparse coding network for hyperspectral and multispectral image fusion

ABSTRACT Although high-resolution hyperspectral images are of great significance in various applications, the manner in which to acquire such images remains a fundamental problem due to sensors’ physical limitations. Fusing high-spatial resolution multispectral images (HR-MSIs) with low-spatial resolution hyperspectral images (LR-HSIs) has become a common approach to increase the spatial resolution of hyperspectral images (HSIs). Various fusion methods have been developed to obtain images with high resolution in both the spatial and spectral domains; the main drawbacks of these methods are time consumption and spectral distortion. With the intention of developing a fast yet accurate image fusion algorithm, this paper is attempted to incorporate the benefits of deep learning and convolutional sparse coding to create a new fusion method. First, the inputs are initialized by solving the Sylvester equation from an up-sampled LR-HSI and an HR-MSI. Second, the learned iterative shrinkage thresholding algorithm is extended to its convolutional version as the central part of the model and recursively learn the sparse priors. Finally, a fused image is achieved by adding the residual to the initialized image. The proposed method is conducted on the widely used datasets (Harvard and iCVL). The experimental results are demonstrated that the proposed approach outperforms several state-of-the-art approaches in terms of quantitative and qualitative perspectives.

Hyperspectral Image Reconstruction Using Multi-scale Fusion Learning

Hyperspectral imaging is a promising imaging modality that simultaneously captures several images for the same scene on narrow spectral bands, and it has made considerable progress in different fields, such as agriculture, astronomy, and surveillance. However, the existing hyperspectral (HS) cameras sacrifice the spatial resolution for providing the detail spectral distribution of the imaged scene, which leads to low-resolution (LR) HS images compared with the common red-green-blue (RGB) images. Generating a high-resolution HS (HR-HS) image via fusing an observed LR-HS image with the corresponding HR-RGB image has been actively studied. Existing methods for this fusing task generally investigate hand-crafted priors to model the inherent structure of the latent HR-HS image, and they employ optimization approaches for solving it. However, proper priors for different scenes can possibly be diverse, and to figure it out for a specific scene is difficult. This study investigates a deep convolutional neural network (DCNN)-based method for automatic prior learning, and it proposes a novel fusion DCNN model with multi-scale spatial and spectral learning for effectively merging an HR-RGB and LR-HS images. Specifically, we construct an U-shape network architecture for gradually reducing the feature sizes of the HR-RGB image (Encoder-side) and increasing the feature sizes of the LR-HS image (Decoder-side), and we fuse the HR spatial structure and the detail spectral attribute in multiple scales for tackling the large resolution difference in spatial domain of the observed HR-RGB and LR-HS images. Then, we employ multi-level cost functions for the proposed multi-scale learning network to alleviate the gradient vanish problem in long-propagation procedure. In addition, for further improving the reconstruction performance of the HR-HS image, we refine the predicted HR-HS image using an alternating back-projection method for minimizing the reconstruction errors of the observed LR-HS and HR-RGB images. Experiments on three benchmark HS image datasets demonstrate the superiority of the proposed method in both quantitative values and visual qualities.