High Resolution Multispectral Images Research Articles

Hyperspectral Image Superresolution via Subspace-Based Deep Prior Regularization

Hyperspectral imaging is able to provide a finer delivery of various material properties than conventional imaging systems. Yet in reality, an optical system can only generate data with high spatial resolution but low spectral one, or vice versa, at video rates. As a result, an issue that fuses low-resolution hyperspectral (LRHS) and high-resolution multispectral (HRMS) images has gained great attention. However, most fusion approaches depend purely on hand-crafted regularizers or data-driven priors, leading to the issues of tricky parameter selection or poor interpretability. In this work, a subspace-based deep prior regularization (SDPR) is proposed to tackle these problems, which takes both hand-crafted regularizer and data-driven prior into account. Specifically, we leverage the spectral correlation of the images and transfer them from the original space to the subspace domain, within which a modified U-net-based deep prior learning network (SDPL-net) is designed for the fusion issue. Moreover, instead of taking the output of SDPL-net directly as the result, we further feed the output back to the model-based optimization. Under such prior regularization, the recovered high-resolution hyperspectral (HRHS) image holds a high consistency to its inherent structure and hence tends to present enhanced reliability and accuracy. Experimental results on simulated and real data reveal that the proposed method excels other state-of-the-art (SOTA) methods in both quantitative and qualitative metrics.

Deep Adaptive Pansharpening via Uncertainty-Aware Image Fusion

Pansharpening is a procedure that fuses high-resolution panchromatic (PAN) images and low-resolution multispectral (LMS) images to derive high-resolution multispectral (HMS) images. Despite its rapid development, most existing pansharpening techniques integrate the information of PAN and LMS invariantly in the spatial dimension, ignoring the uneven spatial dependence of restoring HMS with the aid of PAN information and resulting in ineffective fusion results. In this work, we propose an Uncertainty-aware Adaptive Pansharpening Network (UAPN) that integrates PAN information spatial-variantly to restore LMS information with an uncertainty mechanism. Specifically, we first estimate the epistemic and aleatoric uncertainties together, which model the spatial-variant distributions of restoring the LMS image to the HMS image. Then, we introduce Uncertainty-conditioned Adaptive Convolution (UAC) to adaptively integrate LMS and PAN information, where its parameters are spatially variable by conditioning on the uncertainty estimations. Furthermore, we propose a multi-stage uncertainty-driven loss function to explicitly force the network to concentrate on restoring challenging areas of the LMS image. Extensive experimental results demonstrate the superiority of our UAPN with fewer parameters and flops, outperforming other state-of-the-art methods both qualitatively and quantitatively on multiple satellite datasets. The code is available at https://github.com/keviner1/UAPN..

AWFLN: An Adaptive Weighted Feature Learning Network for Pansharpening

Deep learning (DL)-based pansharpening methods have shown great advantages in extracting spectral-spatial features from multispectral (MS) and panchromatic (PAN) images compared with traditional methods. However, most DL-based methods ignore the local inner connection between the source images and high-resolution MS (HRMS) image, which cannot fully extract spectral-spatial information, and attempt to improve the quality of fusion by increasing the complexity of the network. To solve these problems, a lightweight network based on adaptive weighted feature learning (AWFLN) is proposed for pansharpening. Specifically, a novel detail extraction model is first built by exploring the local relationship between HRMS and source images, thereby improving the accuracy of details and the interpretability of the network. Guided by this model, we then design a residual multiple receptive-field structure to fully extract spectral-spatial features of source images. In this structure, an adaptive feature learning block based on spectral-spatial interleaving attention is proposed to adaptively learn the weights of features and improve the accuracy of the extracted details. Finally, the pansharpened result is obtained by a detail injection model in AWFLN. Numerous experiments are carried out to validate the effectiveness of the proposed method. Compared to traditional and state-of-the-art methods, AWFLN performs the best both subjectively and objectively, with high efficiency. The code is available at https://github.com/yotick/AWFLN.

F-UNet++: Remote Sensing Image Fusion Based on Multipurpose Adaptive Shuffle Attention and Composite Multi-Input Reconstruction Network

The fusion of multispectral (MS) and panchromatic (PAN) images is of great significance for the construction of high-resolution remote sensing images. Because of differences in sensors, no single MS or PAN image can express the complete information of a scene. Therefore, it is a key issue to fuse MS images containing rich spectral content and PAN images with spatial information to construct a high-resolution MS image. In this work, an adaptive shuffle attention (ASA) module and an optimized UNet++ are combined in a fusion-UNet++ (F-UNet++) framework for the problem of MS and PAN image fusion. This ASA module can focus on important information in the mixed domain and adjust the dimensions of tensors. F-UNet++ includes a multiscale feature extraction module, multiscale feature fusion module, and image reconstruction module. The multiscale feature extraction module obtains spectral and spatial information, the multiscale feature fusion module fuses spectral and spatial information, and a composite multi-input image reconstruction module (CMI-UNet++) reconstructs the final image. By combining the ASA attention module, the loss of feature information can be reduced to enhance the fidelity of the spectral and spatial information of the fused image. Experiments show that F-UNet++ is qualitatively and quantitatively superior to current image fusion methods. (The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Josephing/F-UNet</uri> ).

Multiscale Dual-Domain Guidance Network for Pan-Sharpening

The goal of pan-sharpening is to produce a high-spatial-resolution multi-spectral (HRMS) image from a low-spatial-resolution multi-spectral (LRMS) counterpart by super-resolving the LRMS one under the guidance of a texture-rich panchromatic (PAN) image. Existing research has concentrated on using spatial information to generate HRMS images, but has neglected to investigate the frequency domain, which severely restricts the performance improvement. In this work, we propose a novel pan-sharpening approach, named Multi-Scale Dual-Domain Guidance Network (MSDDN) by fully exploring and exploiting the distinguished information in both the spatial and frequency domains. Specifically, the network is inborn with multi-scale U-shape manner and composed by two core parts: a spatial guidance sub-network for fusing local spatial information and a frequency guidance sub-network for fusing global frequency domain information and encouraging dual-domain complementary learning. In this way, the model can capture multi-scale dual-domain information to help it generate high-quality pan-sharpening results. Employing the proposed model on different datasets, the quantitative and qualitative results demonstrate that our method performs appreciatively against other state-of-the-art approaches and comprises a strong generalization ability for real-world scenes. The source code is available at https://github.com/alexhe101/MSDDN.

Hyperspectral Multispectral Image Fusion via Fast Matrix Truncated Singular Value Decomposition

Recently, methods for obtaining a high spatial resolution hyperspectral image (HR-HSI) by fusing a low spatial resolution hyperspectral image (LR-HSI) and high spatial resolution multispectral image (HR-MSI) have become increasingly popular. However, most fusion methods require knowing the point spread function (PSF) or the spectral response function (SRF) in advance, which are uncertain and thus limit the practicability of these fusion methods. To solve this problem, we propose a fast fusion method based on the matrix truncated singular value decomposition (FTMSVD) without using the SRF, in which our first finding about the similarity between the HR-HSI and HR-MSI is utilized after matrix truncated singular value decomposition (TMSVD). We tested the FTMSVD method on two simulated data sets, Pavia University and CAVE, and a real data set wherein the remote sensing images are generated by two different spectral cameras, Sentinel 2 and Hyperion. The advantages of FTMSVD method are demonstrated by the experimental results for all data sets. Compared with the state-of-the-art non-blind methods, our proposed method can achieve more effective fusion results while reducing the fusing time to less than 1% of such methods; moreover, our proposed method can improve the PSNR value by up to 16 dB compared with the state-of-the-art blind methods.

Monitoring Dewatering Fish Spawning Sites in the Reservoir of a Large Hydropower Plant in a Lowland Country Using Unmanned Aerial Vehicles

This paper presents research concerning dewatered areas in the littoral zones of the Kaunas hydropower plant (HPP) reservoir in Lithuania. It is a multipurpose reservoir that is primarily used by two large hydropower plants for power generation. As a result of the peaking operation regime of the Kaunas HPP, the large quantity of water that is subtracted and released into the reservoir by the Kruonis pumped storage hydropower plant (PSP), and the reservoir morphology, i.e., the shallow, gently sloping littoral zone, significant dewatered areas can appear during drawdown operations. This is especially dangerous during the fish spawning period. Therefore, reservoir operation rules are in force that limit the operation of HPPs and secure other reservoir stakeholder needs. There is a lack of knowledge concerning fish spawning locations, how they change, and what areas are dewatered at different stages of HPP operation. This knowledge is crucial for decision-making and efficient reservoir storage management in order to simultaneously increase power generation and protect the environment. Current assessments of the spawning sites are mostly based on studies that were carried out in the 1990s. Surveying fish spawning sites is typically a difficult task that is usually carried out by performing manual bathymetric measurements due to the limitations of sonar in such conditions. A detailed survey of a small (approximately 5 ha) area containing several potential spawning sites was carried out using Unmanned Aerial Vehicles (UAV) equipped with multispectral and conventional RGB cameras. The captured images were processed using photogrammetry and analyzed using various techniques, including machine learning. In order to highlight water and track changes, various indices were calculated and assessed, such as the Normalized Difference Water Index (NDWI), Normalized Difference Vegetation Index (NDVI), Visible Atmospherically Resistant Index (VARI), and Normalized Green-Red Difference Index (NGRDI). High-resolution multispectral images were used to analyze the spectral footprint of aquatic macrophytes, and the possibility of using the results of this study to identify and map potential spawning sites over the entire reservoir (approximately 63.5 km2) was evaluated. The aim of the study was to investigate and implement modern surveying techniques to improve usage of reservoir storage during hydropower plant drawdown operations. The experimental results show that thresholding of the NGRDI and supervised classification of the NDWI were the best-performing methods for the shoreline detection in the fish spawning sites.

DiTBN: Detail Injection-Based Two-Branch Network for Pansharpening of Remote Sensing Images

Pansharpening is one of the main research topics in the field of remote sensing image processing. In pansharpening, the spectral information from a low spatial resolution multispectral (LRMS) image and the spatial information from a high spatial resolution panchromatic (PAN) image are integrated to obtain a high spatial resolution multispectral (HRMS) image. As a prerequisite for the application of LRMS and PAN images, pansharpening has received extensive attention from researchers, and many pansharpening methods based on convolutional neural networks (CNN) have been proposed. However, most CNN-based methods regard pansharpening as a super-resolution reconstruction problem, which may not make full use of the feature information in two types of source images. Inspired by the PanNet model, this paper proposes a detail injection-based two-branch network (DiTBN) for pansharpening. In order to obtain the most abundant spatial detail features, a two-branch network is designed to extract features from the high-frequency component of the PAN image and the multispectral image. Moreover, the feature information provided by source images is reused in the network to further improve information utilization. In order to avoid the training difficulty for a real dataset, a new loss function is introduced to enhance the spectral and spatial consistency between the fused HRMS image and the input images. Experiments on different datasets show that the proposed method achieves excellent performance in both qualitative and quantitative evaluations as compared with several advanced pansharpening methods.

MPFINet: A Multilevel Parallel Feature Injection Network for Panchromatic and Multispectral Image Fusion

The fusion of a high-spatial-resolution panchromatic (PAN) image and a corresponding low-resolution multispectral (MS) image can yield a high-resolution multispectral (HRMS) image, which is also known as pansharpening. Most previous methods based on convolutional neural networks (CNNs) have achieved remarkable results. However, information of different scales has not been fully mined and utilized, and still produces spectral and spatial distortion. In this work, we propose a multilevel parallel feature injection network that contains three scale levels and two parallel branches. In the feature extraction branch, a multi-scale perception dynamic convolution dense block is proposed to adaptively extract the spatial and spectral information. Then, the sufficient multilevel features are injected into the image reconstruction branch, and an attention fusion module based on the spectral dimension is designed in order to fuse shallow contextual features and deep semantic features. In the image reconstruction branch, cascaded transformer blocks are employed to capture the similarities among the spectral bands of the MS image. Extensive experiments are conducted on the QuickBird and WorldView-3 datasets to demonstrate that MPFINet achieves significant improvement over several state-of-the-art methods on both spatial and spectral quality assessments.

Hyperspectral Image Super-Resolution via Deep Spatiospectral Attention Convolutional Neural Networks.

Hyperspectral images (HSIs) are of crucial importance in order to better understand features from a large number of spectral channels. Restricted by its inner imaging mechanism, the spatial resolution is often limited for HSIs. To alleviate this issue, in this work, we propose a simple and efficient architecture of deep convolutional neural networks to fuse a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution HSI (HR-HSI). The network is designed to preserve both spatial and spectral information thanks to a new architecture based on: 1) the use of the LR-HSI at the HR-MSI's scale to get an output with satisfied spectral preservation and 2) the application of the attention and pixelShuffle modules to extract information, aiming to output high-quality spatial details. Finally, a plain mean squared error loss function is used to measure the performance during the training. Extensive experiments demonstrate that the proposed network architecture achieves the best performance (both qualitatively and quantitatively) compared with recent state-of-the-art HSI super-resolution approaches. Moreover, other significant advantages can be pointed out by the use of the proposed approach, such as a better network generalization ability, a limited computational burden, and the robustness with respect to the number of training samples. Please find the source code and pretrained models from https://liangjiandeng.github.io/Projects_Res/HSRnet_2021tnnls.html.

Hierarchical spatio-spectral fusion for hyperspectral image super resolution via sparse representation and pre-trained deep model

Fusing a hyperspectral image (HSI) with a high resolution multispectral image (MSI) has been a highly attractive and effective approach for improving the spatial resolution of HSIs. However, most existing spatio-spectral fusion methods directly accomplish such a fusion process on the desired image scale. This limits the accuracy of the resulting model, especially when large magnification factors are involved. In this paper, a new dual pyramid model is proposed to implement hyperspectral image super resolution, based on a novel hierarchical spatial and spectral fusion method, in an effort to estimate the high resolution image progressively. In particular, an input low resolution HSI is upscaled progressively using a pre-trained deep Laplacian pyramid network, while the corresponding high resolution MSI is down sampled with multiple pyramid layers, forming a dual pyramid model. At each pyramid layer, the HSI and the MSI within the current layer are fused via sparse representation. Systematic qualitative and quantitative evaluations against benchmark datasets demonstrate that this new approach outperforms a number of spatio-spectral fusion based super resolution techniques, achieving outstanding performance over large scale factors.

A hybrid CUDA, OpenMP, and MPI parallel TCA-based domain adaptation for classification of very high-resolution remote sensing images

Domain Adaptation (DA) is a technique that aims at extracting information from a labeled remote sensing image to allow classifying a different image obtained by the same sensor but at a different geographical location. This is a very complex problem from the computational point of view, specially due to the very high-resolution of multispectral images. TCANet is a deep learning neural network for DA classification problems that has been proven as very accurate for solving them. TCANet consists of several stages based on the application of convolutional filters obtained through Transfer Component Analysis (TCA) computed over the input images. It does not require backpropagation training, in contrast to the usual CNN-based networks, as the convolutional filters are directly computed based on the TCA transform applied over the training samples. In this paper, a hybrid parallel TCA-based domain adaptation technique for solving the classification of very high-resolution multispectral images is presented. It is designed for efficient execution on a multi-node computer by using Message Passing Interface (MPI), exploiting the available Graphical Processing Units (GPUs), and making efficient use of each multicore node by using Open Multi-Processing (OpenMP). As a result, an accurate DA technique from the point of view of classification and with high speedup values over the sequential version is obtained, increasing the applicability of the technique to real problems.

Identification of symptoms related to potato Verticillium wilt from UAV-based multispectral imagery using an ensemble of gradient boosting machines

Potato Verticillium wilt is a harmful and widespread disease which causes losses between 20% and 50% of crop production in Colombia. Diagnostics of Verticillium wilt in potato crops is challenging due that the known symptoms are often associated with other diseases. While recent studies have explored the potential of high spatial resolution imagery for detection of diseases and physiological disorders in potato crops, results are not-conclusive yet. Thus, to contribute to improving identification of Verticillium wilt symptoms, we evaluate a machine learning method to classify disease severity from multidate UAV-based multispectral images. It uses an ensemble of gradient boosting machines (GBMs) to differentiate, at grid level, six levels of Verticillim wilt disease severity. The proposed method was tested in a commercial potato crop in Cundinamarca, one of the top potato production regions in Colombia, using multispectral images acquired in four different field campaigns conducted between 99 and 141 days after planting. Accuracy assessment of results indicates that classification of severity levels of Verticillium wilt infection was satisfactory with F1 scores of 82% for the training dataset and 84% for the testing dataset. This study results shows both potential and challenges of using machine learning ensemble methods for identification of Verticillium wilt symptoms in potato crops from high spatial resolution multispectral images.

Integrated remote sensing data and field investigations for geological mapping and structural analysis. The case of SW Tichoukt ridge (Middle Atlas, Morocco)

Boulemane area is one of the important key sectors to understand the evolution of the central Middle Atlas of Morocco. It is a junction area of three Cretaceous/Paleogene unconformable sedimentary basins on the Triassic-Jurassic deposits.This work is a combination of remote sensing results (optical and radar data) supported by field observations to accurate a high-precision geological map of the study area. For this purpose, we used two types of images that are the radar image (Sentinel-1 data) and the optical multispectral images: Landsat 8 OLI (Optical Land Imager) and Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER). The two multispectral images have been spatially enhanced using the panchromatic band of the HRG-2 (High Geometric Resolution) sensor of SPOT 5, allowing the production of high spatial resolution images (5m).The colored compositions that were chosen using OIF (Optimum Index Factor), and PCA (Principal Component Analysis) allowed good discrimination of the main rock. Moreover, ASTER thermal bands have allowed the delimitation of large outcrops of yellow sandstone, conglomerate, phosphate and gypsum based on their thermal emissivity.Structural lineaments are manually extracted from Sentinel-1 radar images after applying directional filters, analyzing Landsat 8 OLI, ASTER high-resolution multispectral images, and interpreting the spatial juxtaposition of lithological features after digitizing the final lithological map.The extracted structural lineaments are grouped into major systems mainly oriented NE-SW, and NW-SE. The NE-SW system is constituted by faults inherited from the Paleozoic basement. The NW-SE transverse system is constituted by faults newly developed during the Middle Liassic.

A Lightweight Multi-Level Information Network for Multispectral and Hyperspectral Image Fusion

The process of fusing the rich spectral information of a low spatial resolution hyperspectral image (LR-HSI) with the spatial information of a high spatial resolution multispectral image (HR-MSI) to obtain an HSI with the spatial resolution of an MSI image is called hyperspectral image fusion (HIF). To reconstruct hyperspectral images at video frame rate, we propose a lightweight multi-level information network (MINet) for multispectral and hyperspectral image fusion. Specifically, we develop a novel lightweight feature fusion model, namely residual constraint block based on global variance fine-tuning (GVF-RCB), to complete the feature extraction and fusion of hyperspectral images. Further, we define a residual activity factor to judge the learning ability of the residual module, thereby verifying the effectiveness of GVF-RCB. In addition, we use cascade cross-level fusion to embed the different spectral bands of the upsampled LR-HSI in a progressive manner to compensate for lost spectral information at different levels and to maintain spatial high frequency information at all times. Experiments on different datasets show that our MINet outperforms the state-of-the-art methods in terms of objective metrics, in particular by requiring only 30% of the running time and 20% of the number of parameters.

The Use of Indices and Modified U-Net Network in Improving the Classification of Planting Structures

It was difficult to accurately obtain crop planting structure by using the spectral information of high spatial resolution and low spatial resolution multispectral images of panchromatic images at the same time. In this paper, we propose a method of planting structure extraction based on indices and an improved U-Net semantic segmentation network. Based on the original band of Landsat-8, we used an image fusion algorithm to highlight the characteristics of vegetation, water, and soil respectively by three indices added, and the improved U-Net network was used to classify the type of planting structure. The results showed that the overall accuracy of classification was more than 91.6%, and the accuracy of crops was up to 93.8%. Automated water extraction index in image fusion effectively improved the classification accuracy. This method could extract a variety of information about planting structures automatically and accurately. It provided theoretical support for adjusting and optimizing regional planting structures.

Classification of Satellite Imagery for Identifying Land-Cover Objects using ECW Compression Images: The Case of Makassar City Area

This paper presents a case study on the effect of lossy compression using the Enhanced Compressed Wavelet (ECW) format on remote sensing image classification. ECW was chosen because it is widely used as a standard format for storing aerial and satellite imagery. The case study was conducted on a high-resolution multispectral Pleiades image taken from an area in Makassar, Indonesia. Image classification is performed using the geographic object-based image analysis method, where a simple linear iterative clustering (SLIC) algorithm is implemented for segmentation before classification. Six land cover categories were selected to validate the classification results: water bodies, trees, rice fields, shrubs, and urban areas. The effect of image compression on classification accuracy is studied by varying the compression ratio. Then the results are compared with the original image. Experimental results prove that compression with ECW format does not have much effect on classification accuracy. Even the Random Forest and Gradient Boosting Machine provide higher accuracy with the compressed image compared to the original image. In addition, it can be concluded that Random Forest is the best classifier with the highest accuracy.

Design and Experiment of High-Resolution Multispectral Polarization Imaging System

This paper addresses the objectives in a complex context: the polarization mathematical and physical model of the basic components of the target/background is constructed by simulation modeling. The image fusion experiment is carried out based on a two-dimensional wavelet transformation. A high-resolution polarization imaging instrument was developed, and static experiments were carried out with polarization, infrared, and visible cameras. The result shows that using polarization imaging detection technology to detect physical evidence targets in complex backgrounds can effectively improve target contrast. It can enrich the target information, improve image quality, and improve target detection accuracy.

Research on the deep learning-based exposure invariant spectral reconstruction method.

The surface spectral reflectance of an object is the key factor for high-fidelity color reproduction and material analysis, and spectral acquisition is the basis of its applications. Based on the theoretical imaging model of a digital camera, the spectral reflectance of any pixels in the image can be obtained through spectral reconstruction technology. This technology can avoid the application limitations of spectral cameras in open scenarios and obtain high spatial resolution multispectral images. However, the current spectral reconstruction algorithms are sensitive to the exposure variant of the test images. That is, when the exposure of the test image is different from that of the training image, the reconstructed spectral curve of the test object will deviate from the real spectral to varying degrees, which will lead to the spectral data of the target object being accurately reconstructed. This article proposes an optimized method for spectral reconstruction based on data augmentation and attention mechanisms using the current deep learning-based spectral reconstruction framework. The proposed method is exposure invariant and will adapt to the open environment in which the light is easily changed and the illumination is non-uniform. Thus, the robustness and reconstruction accuracy of the spectral reconstruction model in practical applications are improved. The experiments show that the proposed method can accurately reconstruct the shape of the spectral reflectance curve of the test object under different test exposure levels. And the spectral reconstruction error of our method at different exposure levels is significantly lower than that of the existing methods, which verifies the proposed method's effectiveness and superiority.

Remote Sensing Image Fusion Based on Morphological Convolutional Neural Networks with Information Entropy for Optimal Scale.

Remote sensing image fusion is a fundamental issue in the field of remote sensing. In this paper, we propose a remote sensing image fusion method based on optimal scale morphological convolutional neural networks (CNN) using the principle of entropy from information theory. We use an attentional CNN to fuse the optimal cartoon and texture components of the original images to obtain a high-resolution multispectral image. We obtain the cartoon and texture components using sparse decomposition-morphological component analysis (MCA) with an optimal threshold value determined by calculating the information entropy of the fused image. In the sparse decomposition process, the local discrete cosine transform dictionary and the curvelet transform dictionary compose the MCA dictionary. We sparsely decompose the original remote sensing images into a texture component and a cartoon component at an optimal scale using the information entropy to control the dictionary parameter. Experimental results show that the remote sensing image fusion method proposed in this paper can effectively retain the information of the original image, improve the spatial resolution and spectral fidelity, and provide a new idea for image fusion from the perspective of multi-morphological deep learning.