Low Resolution Multispectral Image Research Articles

Superpixel guided structure sparsity for multispectral and hyperspectral image fusion over couple dictionary

This paper proposed a hyperspectral (HS) and multispectral (MS) image fusion method based on superpixel guided structure sparsity and couple dictionary (SGSSCD). It is assumed that the pixels in a homogeneous area of MS image are similar due to the consistent spatial consistency. Superpixel technique is used to find the similar pixels in MS image by considering the region homogeneity. Then these pixels naturally share the same atoms in low spectral resolution dictionary. In order to capture the similarity prior, structural sparsity is employed to find more efficient coding of the similar pixels in MS image over low spectral resolution dictionary. Finally, high spatial resolution HS image can be produced by combining the codes of MS image with high spectral resolution dictionary. Besides, the couple dictionary is learned from HS and low spatial resolution MS images to ensure the spectral correspondence, which can further improve the quality of fusion results. The experimental results on different datasets demonstrate the effectiveness of the proposed method when compared with some existing methods.

Convolutional Autoencoder-Based Multispectral Image Fusion

This paper presents a deep learning-based pansharpening method for fusion of panchromatic and multispectral images in remote sensing applications. This method can be categorized as a component substitution method in which a convolutional autoencoder network is trained to generate original panchromatic images from their spatially degraded versions. Low resolution multispectral images are then fed into the trained convolutional autoencoder network to generate estimated high resolution multispectral images. The fusion is achieved by injecting the detail map of each spectral band into the corresponding estimated high resolution multispectral bands. Full reference and no-reference metrics are computed for the images of three satellite datasets. These measures are compared with the existing fusion methods whose codes are publicly available. The results obtained indicate the effectiveness of the developed deep learning-based method for multispectral image fusion.

Remote Sensing Image Sharpening by Integrating Multispectral Image Super-Resolution and Convolutional Sparse Representation Fusion

In remote sensing, it is quite necessary to fuse spectral information of low-resolution multispectral (LRMS) images and spatial information of panchromatic (PAN) images for obtaining high-resolution multispectral (HRMS) images. In this paper, an effective fusion method integrating multispectral (MS) image super-resolution and convolutional sparse representation (CSR) fusion is proposed to make full use of the spatial information of remote sensing images. First, for enhancing the spatial information of LRMS images with suitable sizes, a fast iterative image super-resolution algorithm based on the learned iterative shrinkage and thresholding algorithm (LISTA) is exploited in the first stage. It employs a feed-forward neural network to simplify the solution of sparse coefficients in the process of super-resolution. In the fusion stage, we propose a CSR-based image fusion framework, in which each MS super-resolution image and PAN image is decomposed into a basic layer and a detail layer, then we fuse the basic layers and the detail layers of the images, respectively. This hierarchical fusion strategy guarantees great performance in detail preservation. The experimental results on QuickBird, WorldView-2, and Landsat ETM+ datasets demonstrate that the proposed method outperforms other methods in terms of both objective evaluation and visual effect.

The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior

In this paper, we propose a tensor-based non-convex sparse modeling approach for the fusion of panchromatic and multispectral remote sensing images, and this kind of fusion is generally called pansharpening. We first upsample the low spatial-resolution multispectral image by a classical interpolation method to get an initial upsampled multispectral image. Based on the hyper-Laplacian distribution of errors between the upsampled multispectral image and the ground-truth high resolution multispectral image on gradient domain, we formulate a ℓp(0 < p < 1)-norm term to more reasonably describe the relation of these two datasets. In addition, we also model a tensor-based weighted fidelity term for the panchromatic and low resolution multispectral images, aiming to recover more spatial details. Moreover, total variation regularization is also employed to depict the sparsity of the latent high resolution multispectral image on the gradient domain. For the model solving, we design an alternating direction method of multipliers based algorithm to efficiently solve the proposed model. Furthermore, the involved non-convex ℓp subproblem is handled by an efficient generalized shrinkage/thresholding algorithm. Finally, extensive experiments on many datasets collected by different sensors demonstrate the effectiveness of our method when compared with several state-of-the-art image fusion approaches.

Multispectral Image Super-Resolution via RGB Image Fusion and Radiometric Calibration.

Multispectral imaging is of wide application for its capability in acquiring the spectral information of scenes. Due to hardware limitation, multispectral imaging device usually cannot achieve high-spatial resolution. To address the issue, this paper proposes a multispectral image super-resolution algorithm, referred as SRIF, by fusing the low-resolution multispectral image and the high-resolution (HR) RGB image. It deals with the general circumstance that image intensity is linear to scene radiance for multispectral imaging devices while is nonlinear and unknown for most RGB cameras. The SRIF algorithm first solves the inverse camera response function and a spectral sensitivity function of RGB camera, and establishes the linear relationship between multispectral and RGB images. Then the unknown HR multispectral image is efficiently reconstructed according to the linear image degradation models. Meanwhile, the edge structure of the reconstructed HR multispectral image is kept in accordance with that of the RGB image using a weighted total variation regularizer. The effectiveness of the SRIF algorithm is evaluated on both public datasets and our image set. Experimental results validate that the SRIF algorithm outperforms the state-of-the-arts in terms of both reconstruction accuracy and computational efficiency.

A regularised model-based pan-sharpening method for remote sensing images with local dissimilarities

ABSTRACTCombining the spectral information of a low-resolution multispectral (LRMS) image and the spatial information of a high-resolution panchromatic (HRP) image to generate a high-resolution multispectral (HRMS) image has become an important and interesting issue. Local dissimilarities between the LRMS image and the HRP image affect the performance of the pan-sharpening technique. This paper presents a model-based pan-sharpening method with global and nonlocal spatial similarity regularisers to reduce the effects of the local dissimilarities. The degraded model relating the LRMS image to the unknown HRMS image is employed as the data-fitting term to keep spectral fidelity. Two spatial similarity constraints are utilized to further enhance the spatial resolution of the unknown HRMS image. The first regularisation term is under the assumption that the high-pass component of each HRMS band has the similar geometry structure with the adjusted high-pass component of the HRP image. A modulation matrix is constructed to reduce the contrast differences. Moreover, nonlocal self-similarity characteristic of the high-pass component extracted from each HRMS band is considered as another regulariser, which is an effective structural prior to improve the local spatial quality of the HRMS image. The weights of nonlocal similarity model are learned from the high-pass component of available HRP image. Experiments conducted on QuickBird and IKONOS data validate that the proposed pan-sharpening method can achieve better performance compared with several traditional and state-of-the-art pan-sharpening algorithms in terms of quantitative evaluation and visual analysis.

Remote Sensing Performance Enhancement in Hyperspectral Images.

Hyperspectral images with hundreds of spectral bands have been proven to yield high performance in material classification. However, despite intensive advancement in hardware, the spatial resolution is still somewhat low, as compared to that of color and multispectral (MS) imagers. In this paper, we aim at presenting some ideas that may further enhance the performance of some remote sensing applications such as border monitoring and Mars exploration using hyperspectral images. One popular approach to enhancing the spatial resolution of hyperspectral images is pansharpening. We present a brief review of recent image resolution enhancement algorithms, including single super-resolution and multi-image fusion algorithms, for hyperspectral images. Advantages and limitations of the enhancement algorithms are highlighted. Some limitations in the pansharpening process include the availability of high resolution (HR) panchromatic (pan) and/or MS images, the registration of images from multiple sources, the availability of point spread function (PSF), and reliable and consistent image quality assessment. We suggest some proactive ideas to alleviate the above issues in practice. In the event where hyperspectral images are not available, we suggest the use of band synthesis techniques to generate HR hyperspectral images from low resolution (LR) MS images. Several recent interesting applications in border monitoring and Mars exploration using hyperspectral images are presented. Finally, some future directions in this research area are highlighted.

Remote Sensing Image Fusion Based on Nonlinear IHS and Fast Nonsubsampled Contourlet Transform

The purpose of remote sensing image fusion is to inject the detail image extracted from the panchromatic (PAN) image into the low spatial resolution multispectral (MS) image. A novel remote sensing image fusion method based on fast nonsubsampled contourlet transform (FNSCT) and Nonlinear intensity-hue-saturation (IHS) is presented in this paper. Firstly, the Nonlinear IHS transform is performed on the multispectral image, and then the I-component representing the spatial resolution and the panchromatic image is transformed by NSCT to obtain the low frequency and high frequency. Finally, the coefficients are selected using the improved sum-modified-Laplacian (SML) method and the improved Log-Gabor filter in the low frequency and the high frequency, respectively. Experimental results show that the proposed method is the most advanced fusion method in subjective and objective evaluation, can provide more spatial information, and retain more spectral information compared with several other methods.

Pan-sharpening with a spatial-enhanced variational model

Pan-sharpening is a procedure that fuses a low-resolution multispectral (MS) image and a high-resolution panchromatic (PAN) image into a high-resolution MS image. Existing variational pan-sharpening methods typically consider gradient as the descriptor of the spatial information and enforce this to be consistent between the PAN image and fused image because of their shared scenes. However, the gradient feature is not sufficient to exploit the abundant spatial details. Second-order derivative, which is balanced with the first-order derivative, is introduced to build a variational model such that additional geographic information is extracted from the PAN image. Then corresponding numerical schemes to determine the minimal solution of this model are provided. Through experiments on data from four different satellites, the proposed method is proven to enhance the spatial quality of the fused result and demonstrate stable performance in preserving the spectral information.

Evaluating the effectiveness of fusing remote sensing images with significantly different spatial resolutions for thematic map production

It is very challenging to fuse remote sensing images with significant spatial resolution difference. In this paper, such kind of images are fusing with one posteriori probability model for the purpose of creating thematic maps. The outlined solution consists of parsing the low spatial resolution multispectral images into intensity-hue-saturation triples with spatial and spectral properties separated; the contribution of the old intensity and the panchromatic image is factorized into posteriori probabilities using their wavelet planes under the aid of support vector machine; the wavelet planes are probabilistically weighted to foresee one new pan-sharpened intensity, from which the fused images are produced by inverse IHS. The outlined method along with two representative methods are evaluated with two sets of imagery covering contrasting heterogeneous landscapes for different purposes of road extraction and wheat area map.

Restoration of Pansharpened Images by Conditional Filtering in the PCA Domain

Pansharpening techniques aim at fusing low-resolution multispectral (MS) images and high-resolution panchromatic (PAN) images to produce high-resolution MS images. Despite significant progress in the field, spectral and spatial distortions might still compromise the quality of the results. We introduce a restoration strategy to mitigate artifacts of fused products. After applying the Principal Component Analysis (PCA) transform to a pansharpened image, the chromatic components are filtered conditionally to the geometry of PAN. The structural component is then replaced by the locally histogram-matched PAN for spatial enhancement. Experimental results illustrate the efficiency of the proposed restoration chain.

High-Quality Bayesian Pansharpening

Pansharpening is a process of acquiring a multi-spectral image with high spatial resolution by fusing a low resolution multi-spectral image with a corresponding high resolution panchromatic image. In this paper, a new pansharpening method based on the Bayesian theory is proposed. The algorithm is mainly based on three assumptions: 1) the geometric information contained in the pan-sharpened image is coincident with that contained in the panchromatic image; 2) the pan-sharpened image and the original multi-spectral image should share the same spectral information; and 3) in each pan-sharpened image channel, the neighboring pixels not around the edges are similar. We build our posterior probability model according to above-mentioned assumptions and solve it by the alternating direction method of multipliers. The experiments at reduced and full resolution show that the proposed method outperforms the other state-of-the-art pansharpening methods. Besides, we verify that the new algorithm is effective in preserving spectral and spatial information with high reliability. Further experiments also show that the proposed method can be successfully extended to hyper-spectral image fusion.

Sparse representation over shared coefficients in multispectral pansharpening

The pansharpening process is for obtaining an enhanced image with both high spatial and high spectral resolutions by fusing a panchromatic (PAN) image and a low spatial resolution multispectral (MS) image. Sparse Principal Component Analysis (SPCA) method has been proposed as a pansharpening method, which utilizes sparse coefficients and over-complete dictionaries to represent the remote sensing data. However, this method still has some drawbacks, such as the existence of the block effect. In this paper, based on SPCA, we propose the Sparse over Shared Coefficients (SSC), in which patches are extracted with a sliding distance of 1 pixel from a PAN image, and the MS image shares the sparse representation coefficients trained from the PAN image independently. The fused high-resolution MS image is reconstructed by K-SVD algorithm and iterations, and residual compensation is applied when the down-sampling constraint is not satisfied. The simulated experiment results demonstrate that the proposed SSC method outperforms SPCA and improves the overall effectiveness.

Incorporating an Adaptive Image Prior Model Into Bayesian Fusion of Multispectral and Panchromatic Images

Reconstruction of a sharpened multispectral (MS) image from its coarser measurements, namely the low spatial resolution MS and panchromatic (Pan) images, is a severely illposed inverse problem which requires the definition of an appropriate prior model. This letter incorporates an adaptive Markov random field (MRF)-based prior model into a Bayesian framework to recover the desired MS image. The proposed MRF-based prior model combines the high frequency details of the Pan image with the spectral relation between the bands of the MS image into a single energy function. Consequently, unlike most pansharpening methods, not directly injecting the spatial information of the Pan image into the fused product, the proposed method offers a fused product with minimum spectral distortion, along with perfectly enhanced spatial resolution. Visual and quantitative assessments of the fused products of the proposed method compared to those of some famous pansharpening methods prove the superiority of the proposed method.

Multi-sensor data fusion using NIHS transform and decomposition algorithms

Multi-spectral image fusion is to enhance the details present in multi-spectral bands with the spatial information available in the panchromatic image. Fused images have the effect of spectral distortions and lack of structural similarity. To overcome these limitations, three methods are proposed using intensity, hue, saturation (IHS) and nonlinear IHS (NIHS) transform along with the Dynamic Mode Decomposition (DMD) and 2D-Empirical Mode Decomposition (2D-EMD or IEMD). An intensity plane is calculated from the NIHS transform. The modes are constructed using DMD by considering the variations between the intensity plane computed using NIHS transforms of a low resolution multi-spectral image and a panchromatic image. Similarly, 2D-EMD is also used for image fusion. Modes are subjected to weighted fusion rule to get an intensity plane with spatial and edge information. Finally, the calculated intensity plane is concatenated along with the hue and saturation plane of low-resolution multi-spectral image and transformed into RGB color space. Thus, the fused images have high spatial and edge information on spectral bands. The experiments and its quality assessment assure that proposed methods perform better than the existing methods.

Remote Sensing Image Fusion With Deep Convolutional Neural Network

Remote sensing images with different spatial and spectral resolution, such as panchromatic (PAN) images and multispectral (MS) images, can be captured by many earth-observing satellites. Normally, PAN images possess high spatial resolution but low spectral resolution, while MS images have high spectral resolution with low spatial resolution. In order to integrate spatial and spectral information contained in the PAN and MS images, image fusion techniques are commonly adopted to generate remote sensing images at both high spatial and spectral resolution. In this study, based on the deep convolutional neural network, a remote sensing image fusion method that can adequately extract spectral and spatial features from source images is proposed. The major innovation of this study is that the proposed fusion method contains a two branches network with the deeper structure which can capture salient features of the MS and PAN images separately. Besides, the residual learning is adopted in our network to thoroughly study the relationship between the high- and low-resolution MS images. The proposed method mainly consists of two procedures. First, spatial and spectral features are respectively extracted from the MS and PAN images by convolutional layers with different depth. Second, the feature fusion procedure utilizes the extracted features from the former step to yield fused images. By evaluating the performance on the QuickBird and Gaofen-1 images, our proposed method provides better results compared with other classical methods.

Compensation Details-Based Injection Model for Remote Sensing Image Fusion

Remote sensing image fusion has a potential spectral distortion problem due to the global/local spectral and spatial correlations between panchromatic (PAN) and multispectral (MS) images. To overcome this problem, in this letter, a compensation details-based injection (CDI) fusion model is presented from a new perspective of compensatory learning. In contrast to the traditional method, the two categories of details, namely, the PAN details and the CD, are considered to compensate for the spatial and spectral differences between low-resolution MS (LRMS) and high-resolution MS images. To obtain the CD, a robust sparse representation was employed to calculate the difference between the PAN and MS images during the fusion. The CD combined with the PAN details extracted by a multiscale-guided filter are then injected into the upsampled LRMS image to achieve a fused image. Extensive experiments were undertaken on several image data sets, and the results demonstrate the effectiveness of the proposed CDI method.

Detection of counterfeit banknotes using multispectral images

In this paper, we propose counterfeit banknote detection algorithms using low resolution multispectral images. It has become increasingly difficult to detect professionally produced counterfeit banknotes, so more sophisticated features have had to be implemented in banknotes. However, sensors that are capable of reading these counter-fake features are rather expensive. On the other hand, multispectral images can be used to tackle the counterfeit banknote problem. Recently, multispectral sensors have been developed for ATM applications. We developed efficient counterfeit banknote detection algorithms and the proposed algorithms were tested using 20 different denominations of European Euro (EUR), Indian rupee (INR), and US Dollars (USD). The experimental results show that the proposed methods provided 99.8% classification accuracy for genuine banknotes and 100% detection accuracy for counterfeit banknotes.

A Variational Pan-Sharpening Method Based on Spatial Fractional-Order Geometry and Spectral–Spatial Low-Rank Priors

Pan-sharpening refers to the fusion of a low-resolution (LR) multispectral (MS) image and a high-resolution (HR) panchromatic (PAN) image to obtain an HR MS image (i.e., pan-sharpened MS image). From the point of view of variational complementary data fusion, it becomes an optimization problem with geometry and spectral preserving constraints. In this paper, a novel unified optimizing pan-sharpening model is proposed by integrating a data-generative fidelity term and a compound prior term, which incorporates both spatial fractional-order geometry and spectral–spatial low-rank priors. Specifically, the proposed model consists of three important ingredients: 1) data-generative fidelity term , which models the degradation relationship between the LR and HR MS images to enforce the geometry and spectral preserving constraints; 2) fractional-order total variation-based spatial fractional-order geometry prior term , which especially exploits the spatial fractional-order gradient feature consistence between the PAN and pan-sharpened MS images to transfer the spatial structure information of the PAN image into the pan-sharpened MS image; and 3) weighted nuclear norm-based spectral–spatial low-rank prior term , which exploits the nonlocal patches-based low-rank structural sparsity simultaneously in the pan-sharpened MS image and the LR MS image for further preserving image spatial structures and spectral information. Thus, the main novelty behind the proposed model is an optimizing mechanism by fully taking advantage of the spatial details and texture expressive power of the spatial fractional-order geometry prior as well as the spectral–spatial correlation preserving capacity of the low-rank prior . Finally, the proposed model can be implemented in an alternating direction method of multipliers framework, and thus, an efficient algorithm is presented. To verify the validity, the new proposed method is systematically compared with some state-of-the-art techniques using the Pleiades, GeoEye-1, QuickBird, and WorldView2 satellite data sets in the subjective, objective, and efficiency aspects. The results show that the proposed method performs better than the compared methods in terms of higher spatial and spectral qualities.

A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening

Pan-sharpening is a fundamental and significant task in the field of remote sensing imagery processing, in which high-resolution spatial details from panchromatic images are employed to enhance the spatial resolution of multispectral (MS) images. As the transformation from low spatial resolution MS image to high-resolution MS image is complex and highly nonlinear, inspired by the powerful representation for nonlinear relationships of deep neural networks, we introduce multiscale feature extraction and residual learning into the basic convolutional neural network (CNN) architecture and propose the multiscale and multidepth CNN for the pan-sharpening of remote sensing imagery. Both the quantitative assessment results and the visual assessment confirm that the proposed network yields high-resolution MS images that are superior to the images produced by the compared state-of-the-art methods.