Hyperspectral Image Restoration Research Articles

Hyperspectral image restoration using noise gradient and dual priors under mixed noise conditions

AbstractImages obtained from hyperspectral sensors provide information about the target area that extends beyond the visible portions of the electromagnetic spectrum. However, due to sensor limitations and imperfections during the image acquisition and transmission phases, noise is introduced into the acquired image, which can have a negative impact on downstream analyses such as classification, target tracking, and spectral unmixing. Noise in hyperspectral images (HSI) is modelled as a combination from several sources, including Gaussian/impulse noise, stripes, and deadlines. An HSI restoration method for such a mixed noise model is proposed. First, a joint optimisation framework is proposed for recovering hyperspectral data corrupted by mixed Gaussian‐impulse noise by estimating both the clean data as well as the sparse/impulse noise levels. Second, a hyper‐Laplacian prior is used along both the spatial and spectral dimensions to express sparsity in clean image gradients. Third, to model the sparse nature of impulse noise, an ℓ1 − norm over the impulse noise gradient is used. Because the proposed methodology employs two distinct priors, the authors refer to it as the hyperspectral dual prior (HySpDualP) denoiser. To the best of authors' knowledge, this joint optimisation framework is the first attempt in this direction. To handle the non‐smooth and non‐convex nature of the general ℓp − norm‐based regularisation term, a generalised shrinkage/thresholding (GST) solver is employed. Finally, an efficient split‐Bregman approach is used to solve the resulting optimisation problem. Experimental results on synthetic data and real HSI datacube obtained from hyperspectral sensors demonstrate that the authors’ proposed model outperforms state‐of‐the‐art methods, both visually and in terms of various image quality assessment metrics.

Unsupervised Test-Time Adaptation Learning for Effective Hyperspectral Image Super-Resolution With Unknown Degeneration.

Fusing a low-resolution hyperspectral image (HSI) with a high-resolution (HR) multi-spectral image has provided an effective way for HSI super-resolution (SR). The key lies on inferring the posteriori of the latent (i.e., HR) HSI using an appropriate image prior and the likelihood determined by the degeneration between the latent HSI and the observed images. However, in scenarios with complex imaging environments and various imaging scenes, the prior of HSIs can be prohibitively complicated and the degeneration is often unknown, which causes it difficult to accurately infer the posteriori of each latent HSI. To tackle this problem, we present an unsupervised test-time adaptation learning (UTAL) framework for HSI SR under unknown degeneration. Instead of directly modeling the complicated image prior, it first implicitly learns a content-agnostic prior shared across different images through supervisedly pre-training a mutual-guiding fusion module on extensive synthetic data. Then, it adapts the shared prior to those private characteristics in the latent HSI for posteriori inference through unsupervisedly learning a self-guiding adaptation module and a degeneration estimation network on two observed images in the test phase. Such a two-stage learning scheme models the complicated image prior in a divide-and-conquer manner, which eases the modeling difficulty and improves the prior accuracy. Moreover, the unknown degeneration can be estimated properly. Both of these two advantages empower us to accurately infer the posteriori of the latent HSI, thereby increasing the generalization performance in real applications. Additionally, in order to further mitigate the over-fitting in coping with more challenging cases (e.g., degenerations in both spectral and spatial domains are unknown) and speed up, we propose to meta-train UTAL on extensive synthetic SR tasks and solve it using an alternative optimization strategy such that UTAL learns to produce good generalization performance in real challenging cases with a small number of gradient descent steps. To verify the efficacy of UTAL, we evaluate it on HSI SR tasks with different unknown degenerations as well as some other HSI restoration tasks (e.g., compressive sensing), and report strong results superior to that of existing competitors.

Supervise-Assisted Self-Supervised Deep-Learning Method for Hyperspectral Image Restoration.

Hyperspectral image (HSI) restoration is a challenging research area, covering a variety of inverse problems. Previous works have shown the great success of deep learning in HSI restoration. However, facing the problem of distribution gaps between training HSIs and target HSI, those data-driven methods falter in delivering satisfactory outcomes for the target HSIs. In addition, the degradation process of HSIs is usually disturbed by noise, which is not well taken into account in existing restoration methods. The existence of noise further exacerbates the dissimilarities within the data, rendering it challenging to attain desirable results without an appropriate learning approach. To track these issues, in this article, we propose a supervise-assisted self-supervised deep-learning method to restore noisy degraded HSIs. Initially, we facilitate the restoration network to acquire a generalized prior through supervised learning from extensive training datasets. Then, the self-supervised learning stage is employed and utilizes the specific prior of the target HSI. Particularly, to restore clean HSIs during the self-supervised learning stage from noisy degraded HSIs, we introduce a noise-adaptive loss function that leverages inner statistics of noisy degraded HSIs for restoration. The proposed noise-adaptive loss consists of Stein's unbiased risk estimator (SURE) and total variation (TV) regularizer and fine-tunes the network with the presence of noise. We demonstrate through experiments on different HSI tasks, including denoising, compressive sensing, super-resolution, and inpainting, that our method outperforms state-of-the-art methods on benchmarks under quantitative metrics and visual quality. The code is available at https://github.com/ying-fu/SSDL-HSI.

Latent Diffusion Enhanced Rectangle Transformer for Hyperspectral Image Restoration.

The restoration of hyperspectral image (HSI) plays a pivotal role in subsequent hyperspectral image applications. Despite the remarkable capabilities of deep learning, current HSI restoration methods face challenges in effectively exploring the spatial non-local self-similarity and spectral low-rank property inherently embedded with HSIs. This paper addresses these challenges by introducing a latent diffusion enhanced rectangle Transformer for HSI restoration, tackling the non-local spatial similarity and HSI-specific latent diffusion low-rank property. In order to effectively capture non-local spatial similarity, we propose the multi-shape spatial rectangle self-attention module in both horizontal and vertical directions, enabling the model to utilize informative spatial regions for HSI restoration. Meanwhile, we propose a spectral latent diffusion enhancement module that generates the image-specific latent dictionary based on the content of HSI for low-rank vector extraction and representation. This module utilizes a diffusion model to generatively obtain representations of global low-rank vectors, thereby aligning more closely with the desired HSI. A series of comprehensive experiments were carried out on four common hyperspectral image restoration tasks, including HSI denoising, HSI super-resolution, HSI reconstruction, and HSI inpainting. The results of these experiments highlight the effectiveness of our proposed method, as demonstrated by improvements in both objective metrics and subjective visual quality.

Flex-DLD: Deep Low-Rank Decomposition Model With Flexible Priors for Hyperspectral Image Denoising and Restoration.

Hyperspectral images (HSIs) are composed of hundreds of contiguous waveband images, offering a wealth of spatial and spectral information. However, the practical use of HSIs is often hindered by the presence of complicated noise caused by various factors such as non-uniform sensor response and dark current. Traditional methods for denoising HSIs rely on constrained optimization approaches, where selecting appropriate prior knowledge is critical for achieving satisfactory results. Nevertheless, these traditional algorithms are limited by hand-crafted priors, leaving room for improvement in their denoising performance. Recently, the supervised deep learning technique has emerged as a promising approach for HSI denoising. However, their requirement for paired training data and poor generalization ability on untrained noise distributions pose challenges in practical applications. In this paper, we design a novel algorithm by the synergism of optimization-based methods and deep learning techniques. Specifically, we introduce a plug-and-play Deep Low-rank Decomposition (DLD) model into the optimization framework. Furthermore, we propose an effective mechanism to incorporate traditional prior knowledge into the DLD model. Finally, we provide a detailed analysis of the optimization process and convergence of the proposed method. Empirical evaluations on various tasks, including hyperspectral image denoising and spectral compressive imaging, demonstrate the superiority of our approach over state-of-the-art methods.

Real-Time Compressed Sensing for Joint Hyperspectral Image Transmission and Restoration for CubeSat

Real-Time Compressed Sensing for Joint Hyperspectral Image Transmission and Restoration for CubeSat

Fixed-Point Convergence of Multi-block PnP ADMM and Its Application to Hyperspectral Image Restoration

Fixed-Point Convergence of Multi-block PnP ADMM and Its Application to Hyperspectral Image Restoration

Attention-Guided Low-Rank Tensor Completion.

Low-rank tensor completion (LRTC) aims to recover missing data of high-dimensional structures from a limited set of observed entries. Despite recent significant successes, the original structures of data tensors are still not effectively preserved in LRTC algorithms, yielding less accurate restoration results. Moreover, LRTC algorithms often incur high computational costs, which hinder their applicability. In this work, we propose an attention-guided low-rank tensor completion (AGTC) algorithm, which can faithfully restore the original structures of data tensors using deep unfolding attention-guided tensor factorization. First, we formulate the LRTC task as a robust factorization problem based on low-rank and sparse error assumptions. Low-rank tensor recovery is guided by an attention mechanism to better preserve the structures of the original data. We also develop implicit regularizers to compensate for modeling inaccuracies. Then, we solve the optimization problem by employing an iterative technique. Finally, we design a multistage deep network by unfolding the iterative algorithm, where each stage corresponds to an iteration of the algorithm; at each stage, the optimization variables and regularizers are updated by closed-form solutions and learned deep networks, respectively. Experimental results for high dynamic range imaging and hyperspectral image restoration show that the proposed algorithm outperforms state-of-the-art algorithms.

A Joint Framework for Underwater Hyperspectral Image Restoration and Target Detection With Conditional Diffusion Model

A Joint Framework for Underwater Hyperspectral Image Restoration and Target Detection With Conditional Diffusion Model

An Improved Plug and Play Tensor Fiber Rank Constrained Hyperspectral Image Restoration

An Improved Plug and Play Tensor Fiber Rank Constrained Hyperspectral Image Restoration

Hyperspectral image restoration based on color superpixel segmentation

Hyperspectral images (HSI) are often degraded by various types of noise during the acquisition process, such as Gaussian noise, impulse noise, dead lines and stripes, etc. Recently, there exists a growing attenrion on low-rank matrix/tensor-based methods for HSI data restoration, assuming that the overall data is low-rank. However, the assumption of overall low-rankness often proves inaccurate due to the spatially heterogeneous local similarity characteristics of HSI. Traditional cube-based methods involve dividing the HSI into fixed-size cubes. However, using fixed-size cubes does not provide flexible coverage of locally similar regions at varying scales. Inspired by superpixel segmentation, this paper proposes the Shrink Low-rank Super-tensor (SLRST) approach for HSI recovery. Instead of using fixed-size cubes, SLRST employs a size-adaptive super-tensor. The proposed approach is effectively solved using the Alternating Direction Method of Multipliers (ADMM). Numerical experiments on HSI data verify that the proposed method outperforms other competing methods.

Hyperspectral fruit image restoration using non-convex optimization

Hyperspectral (HS) imaging captured at low proximity ranges has become an effective method for assessing the quality and safety of food products in recent years. The majority of Hyperspectral cameras are push-broom type, which often contributes random noise and stripes to the spectral images. As a result, the quality of the collected HS images and the precision of the subsequent processing will both suffer. Despite the fact that there are several denoising techniques that have been studied and recorded in the literature, they frequently fail to fully restore the original HS images when there are strong stripe noises and other intricate mixed noises. In this study, we introduce a new method based on tensor logarithmic Schatten-p norm minimization with anisotropic spatial spectral total variation (ASSTV) regularization for simultaneous denoising and destriping of line scan-based hyperspectral images. Tensor logarithmic Schatten-p norm improves denoising performance by deleting the smaller singular values while protecting the larger singular values. Through the application of ASSTV, stripes and Gaussian noise are eliminated while maintaining the spatial–spectral smoothness of HS images and the directional characteristics of the stripes. The augmented Lagrange multiplier (ALM) approach is used to address the ensuing optimization problem. The quantitative and visual assessments reveal that the suggested technique has improved performance than its counterparts.

DnRCNN: Deep Recurrent Convolutional Neural Network for HSI Destriping.

In spite of achieving promising results in hyperspectral image (HSI) restoration, deep-learning-based methodologies still face the problem of spectral or spatial information loss due to neglecting the inner correlation of HSI. To address this issue, we propose an innovative deep recurrent convolution neural network (DnRCNN) model for HSI destriping. To the best of our knowledge, this is the first study on HSI destriping from the perspective of inner band and interband correlation explorations with the recurrent convolution neural network. In the novel DnRCNN, a selective recurrent memory unit (SRMU) is designed to respectively extract the correlative features involved in spectral and spatial domains. Moreover, an innovative recurrent fusion (RF) strategy incorporated with group concatenation is further proposed to remove strip noise and preserve scene details using the complementary features from SRMU. Experimental results on extensive HSI datasets validated that the proposed method achieves a new state-of-the-art (SOTA) HSI destriping performance.

Low-rank tensor completion with spatial-spectral consistency for hyperspectral image restoration

Low-rank tensor completion with spatial-spectral consistency for hyperspectral image restoration

Tensor Compressive Sensing Fused Low-Rankness and Local-Smoothness

A plethora of previous studies indicates that making full use of multifarious intrinsic properties of primordial data is a valid pathway to recover original images from their degraded observations. Typically, both low-rankness and local-smoothness broadly exist in real-world tensor data such as hyperspectral images and videos. Modeling based on both properties has received a great deal of attention, whereas most studies concentrate on experimental performance, and theoretical investigations are still lacking. In this paper, we study the tensor compressive sensing problem based on the tensor correlated total variation, which is a new regularizer used to simultaneously capture both properties existing in the same dataset. The new regularizer has the outstanding advantage of not using a trade-off parameter to balance the two properties. The obtained theories provide a robust recovery guarantee, where the error bound shows that our model certainly benefits from both properties in ground-truth data adaptively. Moreover, based on the ADMM update procedure, we design an algorithm with a global convergence guarantee to solve this model. At last, we carry out experiments to apply our model to hyperspectral image and video restoration problems. The experimental results show that our method is prominently better than many other competing ones. Our code and Supplementary Material are available at https://github.com/fsliuxl/cs-tctv.

Probability-weighted tensor robust PCA with CP decomposition for hyperspectral image restoration

This paper presents a novel probability-weighted tensor robust principal component analysis (TRPCA) method based on CANDECOMP/PARAFAC decomposition (CPD) for hyperspectral image (HSI) restoration with low computational complexity from the coexistence of dense noise and sparse outliers. In the proposed method, via the CPD property, the tensor nuclear norm (TNN) optimization object is replaced as the CPD factor matrix of HSI with lower dimensions without losing low-rank property, so as to the computational complexity could be reduced during optimizing TNN in TRPCA model. To demonstrate the different effects of noise and outliers on HSI restoration, two weighted sets are defined with probability by the prior knowledge of the occurrence of outliers, which are used to analyze the probability whether the HSI is corrupted by dense noise or outliers. Via the probability theory, information entropy is employed to measure the uncertainty of noise and outlier occurrence in received data, so that to demonstrate their difference impacts on HSI restoration to improve the recovery accuracy. Theoretical analysis and simulation results show that the proposed method has lower computational complexity and better restoration performance compared with other related methods.

A survey on hyperspectral image restoration: from the view of low-rank tensor approximation

A survey on hyperspectral image restoration: from the view of low-rank tensor approximation

Robust Superpixel Segmentation for Hyperspectral-Image Restoration

Hyperspectral-image (HSI) restoration plays an essential role in remote sensing image processing. Recently, superpixel segmentation-based the low-rank regularized methods for HSI restoration have shown outstanding performance. However, most of them simply segment the HSI according to its first principal component, which is suboptimal. In this paper, integrating the superpixel segmentation with principal component analysis, we propose a robust superpixel segmentation strategy to better divide the HSI, which can further enhance the low-rank attribute of the HSI. To better employ the low-rank attribute, the weighted nuclear norm by three types of weighting is proposed to efficiently remove the mixed noise in degraded HSI. Experiments conducted on simulated and real HSI data verify the performance of the proposed method for HSI restoration.

Hyperspectral image restoration via hybrid smoothness regularized auto-weighted low-rank tensor ring factorization

Restoration of hyperspectral images (HSI) is a crucial step in many potential applications as a preprocessing step. Recently, low-rank tensor ring factorization was applied for HSI reconstruction, which has high-order tensors’ powerful and generalized representation ability. Although low-rank TR-based approaches with nuclear norm regularization achieved successful results for restoring hyperspectral images, there is still room for improved tensor low-rank approximation. In this article, we propose a novel Auto-weighted low-rank Tensor Ring Factorization with Hybrid Smoothness regularization (ATRFHS) for mixed noise removal in HSI. Nonlocal Cuboid Tensorization (NCT) is leveraged to transform HSI data into high-order tensors. TR factorization using latent factors rank minimization removes the mixed noise in HSI data. To highlight nuclear norms of factor tensors differently effective, an auto-weighted strategy is employed to reduce the more prominent factors while shrinking the smaller ones. A hybrid regularization combining total variation (TV) and phase congruency (PC) is incorporated into a low-rank tensor ring factorization model for the HSI noise removal problem. This efficient combination yields sharper edge preservation and resolves this weakness of existing pure TV regularization. Moreover, we develop an efficient algorithm for solving the resulting optimization problem using the framework of alternating minimization. Extensive experimental results demonstrate that our proposed method can significantly outperform existing approaches for mixed noise removal in HSI. The proposed algorithm is validated on synthetic and natural HSI data.

A Novel Tensor-Based Hyperspectral Image Restoration Method With Low-Rank Modeling in Gradient Domains

The hyperspectral image (HSI) is easily contaminated by various kinds of mixed noise (such as Gaussian noise, impulse noise, stripes, and deadlines) during the process of data acquisition and conversion, which significantly affect the quality and applications of HSI. As an important and effective scheme for the quality improvement of HSI, the HSI restoration problem aims to recover a clean HSI from the noisy HSI with mixed noise. Thus, based on the tensor modeling of HSI, we propose a novel tensor-based HSI restoration model with low-rank modeling in gradient domains in a unified tensor representation framework in this article. First, for the spectral low-rank modeling of HSI in spectral gradient domain, we particularly exploit the low-rank property of spectral gradient, and propose the spectral gradient-based weighted nuclear norm low-rank prior term. Second, for the spatial-mode low-rank modeling of HSI in spatial gradient domain, we particularly exploit the low-rank property of spatial gradient tensors via the discrete Fourier transform, and propose the spatial gradient-based tensor nuclear norm low-rank prior term. Then, we use the alternative direction method of multipliers to solve the proposed model. Finally, the restoration results on both the simulated and real HSI datasets demonstrate that the proposed method is superior to many state-of-the-art methods in the aspects of visual and quantitative comparisons.