Hyperspectral Image Datasets Research Articles

Surgical optomics: hyperspectral imaging and deep learning towards precision intraoperative automatic tissue recognition-results from the EX-MACHYNA trial.

Hyperspectral imaging (HSI), combined with machine learning, can help to identify characteristic tissue signatures enabling automatic tissue recognition during surgery. This study aims to develop the first HSI-based automatic abdominal tissue recognition with human data in a prospective bi-center setting. Data were collected from patients undergoing elective open abdominal surgery at two international tertiary referral hospitals from September 2020 to June 2021. HS images were captured at various time points throughout the surgical procedure. Resulting RGB images were annotated with 13 distinct organ labels. Convolutional Neural Networks (CNNs) were employed for the analysis, with both external and internal validation settings utilized. A total of 169 patients were included, 73 (43.2%) from Strasbourg and 96 (56.8%) from Verona. The internal validation within centers combined patients from both centers into a single cohort, randomly allocated to the training (127 patients, 75.1%, 585 images) and test sets (42 patients, 24.9%, 181 images). This validation setting showed the best performance. The highest true positive rate was achieved for the skin (100%) and the liver (97%). Misclassifications included tissues with a similar embryological origin (omentum and mesentery: 32%) or with overlaying boundaries (liver and hepatic ligament: 22%). The median DICE score for ten tissue classes exceeded 80%. To improve automatic surgical scene segmentation and to drive clinical translation, multicenter accurate HSI datasets are essential, but further work is needed to quantify the clinical value of HSI. HSI might be included in a new omics science, namely surgical optomics, which uses light to extract quantifiable tissue features during surgery.

Enhancing Semi-Supervised Few-Shot Hyperspectral Image Classification via Progressive Sample Selection

Hyperspectral images (HSIs) provide valuable spatial–spectral information for ground analysis. However, in few-shot (FS) scenarios, the limited availability of training samples poses significant challenges in capturing the sample distribution under diverse environmental conditions. Semi-supervised learning has shown promise in exploring the distribution of unlabeled samples through pseudo-labels. Nonetheless, FS HSI classification encounters the issue of high intra-class spectral variability and inter-class spectral similarity, which often lead to the diffusion of unreliable pseudo-labels during the iterative process. In this paper, we propose a simple yet effective progressive pseudo-label selection strategy that leverages the spatial–spectral consistency of HSI pixel samples. By leveraging spatially aligned ground materials as connected regions with the same semantic and similar spectrum, pseudo-labeled samples were selected based on round-wise confidence scores. Samples within both spatially and semantically connected regions of FS samples were assigned pseudo-labels and joined subsequent training rounds. Moreover, considering the spatial positions of FS samples that may appear in diverse patterns, to fully utilize unlabeled samples that fall outside the neighborhood of FS samples but still belong to certain connected regions, we designed a matching active learning approach for expert annotation based on the temporal confidence difference. We identified samples with the highest training value in specific regions, utilizing the consistency between predictive labels and expert labels to decide whether to include the region or the sample itself in the subsequent semi-supervised iteration. Experiments on both classic and more recent HSI datasets demonstrated that the proposed base model achieved SOTA performance even with extremely rare labeled samples. Moreover, the extended version with active learning further enhances performance by involving limited additional annotation.

Enhanced Wild Horse Optimizer with Cauchy Mutation and Dynamic Random Search for Hyperspectral Image Band Selection

The high dimensionality of hyperspectral images (HSIs) brings significant redundancy to data processing. Band selection (BS) is one of the most commonly used dimensionality reduction (DR) techniques, which eliminates redundant information between bands while retaining a subset of bands with a high information content and low noise. The wild horse optimizer (WHO) is a novel metaheuristic algorithm widely used for its efficient search performance, yet it tends to become trapped in local optima during later iterations. To address these issues, an enhanced wild horse optimizer (IBSWHO) is proposed for HSI band selection in this paper. IBSWHO utilizes Sobol sequences to initialize the population, thereby increasing population diversity. It incorporates Cauchy mutation to perturb the population with a certain probability, enhancing the global search capability and avoiding local optima. Additionally, dynamic random search techniques are introduced to improve the algorithm search efficiency and expand the search space. The convergence of IBSWHO is verified on commonly used nonlinear test functions and compared with state-of-the-art optimization algorithms. Finally, experiments on three classic HSI datasets are conducted for HSI classification. The experimental results demonstrate that the band subset selected by IBSWHO achieves the best classification accuracy compared to conventional and state-of-the-art band selection methods, confirming the superiority of the proposed BS method.

“Image-Spectral” fusion monitoring of small cotton samples nitrogen content based on improved deep forest

Accurate monitoring of nitrogen nutrition is crucial for improving cotton yield and quality, as well as the ecological environment. The mainstream method for monitoring nutrition is to establish traditional machine learning (ML) models using a single data source. However, this approach has limitations such as limited access to feature information, model-fitting problems, and limited generalization. Deep learning (DL), on the other hand, has shown promise in complex nonlinear modeling tasks due to its flexible structure. However, it has its limitations, such as the fact that agronomic sample collection and testing are usually labor- and material-intensive, resulting in sample sizes that are too small to meet its training conditions. Therefore, there is an urgent need for DL models that can effectively integrate features from multiple data sources and accurately monitor crop nitrogen content, especially in scenarios with small samples. In this study, we conducted indoor pot experiments using the cotton variety Xinluzao 53 and subjected it to six nitrogen treatments. The data sources for our analysis included hyperspectral and digital images of the cotton leaves. To enhance representation learning capabilities, we enriched the multi-class base learners within each layer of the deep forest (DF) model and introduced skip connections. These enhancements improved the quality of inversion for both hyperspectral and digital image datasets. We then developed image-spectral fusion models, which combined the DF structure with stacking ensemble learning. Our focus was on three levels of fusion: feature-level fusion, decision-level fusion, and secondary decision-level fusion. This approach aimed to further enhance the accuracy and stability of nitrogen content inversion. The DF model satisfied the training condition for small samples. Compared to traditional ML algorithms and the original DF algorithm, the improved DF model achieved an increase in validation set R2 of 13.4–28.5% and 10.9–14.9%, respectively. These findings highlight the enhanced accuracy and stability of the improved DF model. Additionally, compared to the optimal inversion model using two single data sources, the “Image-Spectral” three-level fusion models exhibited improvements in validation set R2 of 8.6–9.3%, 10.5–11.2%, and 11.8–12.5% for feature-level, decision-level, and secondary decision-level fusion, respectively. The improved DF and three-level fusion model collectively contributed to the increased accuracy of cotton nitrogen content inversion. Among these models, the secondary decision-level fusion model demonstrated the most marked improvement. This methodology provides valuable insights into monitoring crop phenotypic parameters in situations with limited sample sizes.

Joint negative–positive-learning based sample reweighting for hyperspectral image classification with label noise

Deep neural networks (DNNs) have been widely used for hyperspectral image (HSI) classification. However, the superior performance of DNNs requires accurately labeled samples. In various applications, label noise in large-scale HSI data is often unavoidable and adversely affects the performance of a classifier. Therefore, establishing a robust classification model from an existing DNN classifier for the HSI dataset with label noise is a significant and challenging problem. In this report, we propose a meta-learning reweighting framework based on a joint positive and negative learning (JPNL) method which can adaptively reweight labeled samples to improve the robustness of the classification model. Experimental results on two commonly used hyperspectral imaging (HSI) datasets contaminated with label noise demonstrate significant improvements in classification performance and robustness achieved by the proposed framework, as compared to existing classifiers.

Multispectral and hyperspectral images fusion based on subspace representation and nonlocal low-rank regularization

ABSTRACT Multispectral image (MSI) and hyperspectral image (HSI) fusion is a popular method for HSI super-resolution reconstruction (HSI-SR). MSI-HSI fusion problem is ill-posed and demands several image priors or regularization terms to solve accurately, which is a challenging issue. In this paper, we propose an MSI-HSI fusion model via subspace representation and nonlocal low-rank regularization (SRNLRR). SRNLRR model incorporates the global spectral correlations and spatial nonlocal similarities of HSI to improve the fusion results, where the priors complement each other. First, we use the mode-n tensor-matrix product to project latent high spatial resolution HSI (HR-HSI) into spectral subspace, which can capture spectral low-rankness and reduce computational complexity. Then, based on low-rank representation (LRR) and nonlocal processing strategy, we design a spatial nonlocal LRR regularization (spa-NLRR) and a spectral global LRR regularization (spe-GLRR). These two regularizations analyze the spatial nonlocal similarities and spectral global correlations from intermediate-level vision. Finally, we use the residual regularization program to obtain more image information and input it into the fusion model. We use alternate minimization (AM) methods to optimize the SRNLRR model and employ the alternating direction method (ADM) on spatial/spectral LRR learning. Comparison experiments between the SRNLRR method and six advanced methods on three HSI datasets indicate the superiority of our method.

Adaptive Shadow Compensation Method in Hyperspectral Images via Multi-Exposure Fusion and Edge Fusion

Shadows in hyperspectral images lead to reduced spectral intensity and changes in spectral characteristics, significantly hindering analysis and applications. However, current shadow compensation methods face the issue of nonlinear attenuation at different wavelengths and unnatural transitions at the shadow boundary. To address these challenges, we propose a two-stage shadow compensation method based on multi-exposure fusion and edge fusion. Initially, shadow regions are identified through color space conversion and an adaptive threshold. The first stage utilizes multi-exposure, generating a series of exposure images through adaptive exposure coefficients that reflect spatial shadow intensity variations. Fusion weights for exposure images are determined based on exposure, contrast, and spectral variance. Then, the exposure sequence and fusion weights are constructed as Laplacian pyramids and Gaussian pyramids, respectively, to obtain a weighted fused exposure sequence. In the second stage, the previously identified shadow regions are smoothly reintegrated into the original image using edge fusion based on the p-Laplacian operator. To further validate the effectiveness and spectral fidelity of our method, we introduce a new hyperspectral image dataset. Experimental results on the public dataset and proposed dataset demonstrate that our method surpasses other mainstream shadow compensation methods.

Cross-Scene Hyperspectral Image Classification Based on Graph Alignment and Distribution Alignment

A domain alignment-based hyperspectral image (HSI) classification method was designed to address the heterogeneity in resolution and band between the source domain and target domain datasets of cross-scene hyperspectral images, as well as the resulting reduction in common features. Firstly, after preliminary feature extraction, perform two domain alignment operations: image alignment and distribution alignment. Image alignment aims to align hyperspectral images of different bands or time points, ensuring that they are within the same spatial reference framework. Distribution alignment adjusts the distribution of features of samples of different categories in the feature space to reduce the distribution differences of the same type of features between two domains. Secondly, adjust the consistency of the two alignment methods to ensure that the features obtained through different alignment methods exhibit consistency in the feature space, thereby improving the comparability and reliability of the features. In addition, this method considers multiple losses in the model from different perspectives and makes comprehensive adjustments through a unified optimization process to more comprehensively capture and utilize the correlation information between data. Experimental results on Houston 2013 and Houston 2018 datasets can improve the hyperspectral prediction performance between datasets with different resolutions and bands, effectively solving the problems of high cost and limited training samples in HSI labeling and significantly improving cross-scene HSI classification performance.

Spatial Invariant Tensor Self-Representation Model for Hyperspectral Anomaly Detection.

With the development of hyperspectral imaging technology, the hyperspectral anomaly has attracted considerable attention due to its significant role in many applications. Hyperspectral images (HSIs) with two spatial dimensions and one spectral dimension are intrinsically three-order tensors. However, most of the existing anomaly detectors were designed after converting the 3-D HSI data into a matrix, which destroys the multidimension structure. To solve this problem, in this article, we propose a spatial invariant tensor self-representation (SITSR) hyperspectral anomaly detection algorithm, which is derived based on the tensor-tensor product (t-product) to preserve the multidimension structure and achieve a comprehensive description of the global correlation of HSIs. Specifically, we exploit the t-product to integrate spectral information and spatial information, and the background image of each band is represented as the sum of the t-product of all bands and their corresponding coefficients. Considering the directionality of the t-product, we utilize two tensor self-representation methods with different spatial modes to obtain a more balanced and informative model. To depict the global correlation of the background, we merge the unfolding matrices of two representative coefficients and constrain them to lie in a low-dimensional subspace. Moreover, the group sparsity of anomaly is characterized by l2.1.1 norm regularization to promote the separation of background and anomaly. Extensive experiments conducted on several real HSI datasets demonstrate the superiority of SITSR compared with state-of-the-art anomaly detectors.

Factor annealing decoupling compositional training method for imbalanced hyperspectral image classification

AbstractDue to differences in the quantity and size of observed targets, hyperspectral images are characterized by class imbalance. The standard deep learning classification model training scheme optimizes the overall classification error, which may lead to performance imbalance between classes in hyperspectral image classification frameworks. Therefore, a novel factor annealing decoupling compositional training method is proposed in this paper. Without requiring resampling or reweighting, it implicitly modulates the training process, so standard models can sufficiently learn the representation of the minority classes and further be trained as robust classifiers. Specifically, the label‐distribution‐aware margin loss is combined with the error‐rate‐based cross‐entropy loss via combination factor, which considers both imbalanced data representation learning and classifier overall performance. Then, a factor annealing optimization training scheme is designed to adjust the combination factor, which solves the stage division problem of two‐stage decoupling learning. Experimental results on two hyperspectral image datasets demonstrate that, as compared with other competing approaches, the proposed method can continuously and stably optimize the model parameters, achieving improvements in class average metrics and difficult classes without affecting overall classification performance.

Detailed Land Use Classification in a Rare Earth Mining Area Using Hyperspectral Remote Sensing Data for Sustainable Agricultural Development

In China, ion-adsorbing rare earth minerals are mainly located in the southern hilly areas and are important strategic resources. Extensive long-term mining has severely damaged the land cover in mining areas, caused soil pollution and terrain fragmentation, disrupted the balance between mining and agriculture, severely restricted agricultural development, and affected ecological development. Precise and detailed classification of land use within mining areas is crucial for monitoring the sustainable development of agricultural ecology in these areas. In this study, we leverage the high spatial and high spectral resolution characteristics of the Zhuhai-1 (OHS) hyperspectral image datasets. We create four types of datasets based on spectral, vegetation, red edge, and texture characteristics. These datasets are optimized for multifaceted features, considering the complex land use scenario in rare earth mining areas. Additionally, we design seven optimal combination schemes for features. This is performed to examine the impact of different schemes on land use classification in rare earth mining areas and the accuracy of identifying agricultural land classes from broken blocks. The results show that (1) the inclusion of texture features has the most obvious effect on the overall classification accuracy; (2) the red edge feature has the worst effect on improving the overall accuracy of the surface classification; however, it has a prominent effect on the identification of agricultural lands such as farmland, orchards, and reclaimed vegetation; and (3), following the combination of various optimization features, the land use classification yielded the highest overall accuracy, at 88.16%. Furthermore, the comprehensive identification of various agricultural land classes, including farmland, orchards, and greenhouse vegetables, yielded the most desirable outcomes. The research results not only highlight the advantages of hyperspectral images for complex terrain classification and recognition but also address the previous limitations in the application of hyperspectral datasets over wide mining areas. Additionally, the results underscore the reliability of feature selection methods in reducing information redundancy and improving classification accuracy. The proposed feature selection combination, based on OHS hyperspectral datasets, offers technical support and guidance for the detailed classification of complex land use in mining areas and the accurate monitoring of agroecological environments.

Classification of Microscopic Hyperspectral Images of Blood Cells Based on Lightweight Convolutional Neural Network

Hyperspectral imaging has emerged as a novel imaging modality in the medical field, offering the ability to acquire images of biological tissues while simultaneously providing biochemical insights for in-depth tissue analysis. This approach facilitates early disease diagnosis, presenting advantages over traditional medical imaging techniques. Addressing challenges such as the computational burden of existing convolutional neural networks (CNNs) and imbalances in sample data, this paper introduces a lightweight GhostMRNet for the classification of microscopic hyperspectral images of human blood cells. The proposed model employs Ghost Modules to replace conventional convolutional layers and a cascading approach with small convolutional kernels for multiscale feature extraction, aiming to enhance feature extraction capabilities while reducing computational complexity. Additionally, an SE (Squeeze-and-Excitation) module is introduced to selectively allocate weights to features in each channel, emphasizing informative features and efficiently achieving spatial–spectral feature extraction in microscopic hyperspectral imaging. We evaluated the performance of the proposed GhostMRNet and compared it with other state-of-the-art models using two real medical hyperspectral image datasets. The experimental results demonstrate that GhostMRNet exhibits a superior performance, with an overall accuracy (OA), average accuracy (AA), and Kappa coefficient reaching 99.965%, 99.565%, and 0.9925, respectively. In conclusion, the proposed GhostMRNet achieves a superior classification performance at a smaller computational cost, thereby providing a novel approach for blood cell detection.

An adaptive evolutionary-reinforcement learning algorithm for hyperspectral band selection

Band selection is one of the most important considerations for hyperspectral imaging (HSI) datasets that involves selecting fewer bands with higher classification accuracy. As a combinatorial optimization problem, evolutionary algorithm has been applied in the field, but the optimization ability and number of selected bands are limited with the control of exploration and exploitation. In this paper, an adaptive evolutionary-reinforcement learning algorithm (ERLA) is proposed for band selection of HSI datasets, only a suitable phase contributes to updating the corresponding agents in each iteration, and the coding length is adaptively lessened with the potential solution space. In addition, reinforcement learning is used to choose a feasible action in each iteration, and the unselected bands are removed from the candidate band combination in the previous state. A simulation study is conducted on four public HSI datasets, where the proposed framework has delivered an overall accuracy of 98% for 12% of bands in Chikusei dataset, 92% for 11% in KSC dataset, 93% against 19% for Xiongan dataset, and 97% in Longkou dataset with 10% bands retained, demonstrating that the proposed framework outperforms other approaches with several targets, and that the number of selected bands is obviously decreased with satisfactory classification accuracy.

Data-Driven Analysis of High-Resolution Hyperspectral Image Data Sets through Nanoscale Capacitance–Voltage Measurements to Visualize Ferroelectric Domain Dynamics

Data-Driven Analysis of High-Resolution Hyperspectral Image Data Sets through Nanoscale Capacitance–Voltage Measurements to Visualize Ferroelectric Domain Dynamics

Pixel-associated autoencoder for hyperspectral anomaly detection

Autoencoders (AEs) are central to hyperspectral anomaly detection, given their impressive efficacy. However, the current methodologies often neglect the global pixel similarity of the hyperspectral image (HsI), thereby limiting reconstruction accuracy. This study introduces an innovative pixel-associated AE approach that leverages pixel associations to augment hyperspectral anomaly detection. First, a dictionary construction methodology was introduced based on superpixel distance estimation to construct distinct dictionaries for background and local anomalies. Second, to recognize pixel similarities, the similarity metric of each pixel from the original HsI to the background dictionary and to the local anomaly dictionary was employed as the AE network input in lieu of the original HsI. Third, a dual hidden-layer feature similarity constraint network was proposed to enhance the reconstruction error of background and anomaly targets. Finally, the reconstruction error was utilized to score the anomaly target. The proposed method was benchmarked against other state-of-the-art techniques using synthetic and real HsI datasets to assess its effectiveness. The experimental results demonstrated the superior performance of the proposed method, outperforming the alternatives.

Generative Adversarial Network and Mutual-Point Learning Algorithm for Few-Shot Open-Set Classification of Hyperspectral Images

Existing approaches addressing the few-shot open-set recognition (FSOSR) challenge in hyperspectral images (HSIs) often encounter limitations stemming from sparse labels, restricted category numbers, and low openness. These limitations compromise stability and adaptability. In response, an open-set HSI classification algorithm based on data wandering (DW) is introduced in this research. Firstly, a K-class classifier suitable for a closed set is trained, and its internal encoder is leveraged to extract features and estimate the distribution of known categories. Subsequently, the classifier is fine-tuned based on feature distribution. To address the scarcity of samples, a sample density constraint based on the generative adversarial network (GAN) is employed to generate synthetic samples near the decision boundary. Simultaneously, a mutual-point learning method is incorporated to widen the class distance between known and unknown categories. In addition, a dynamic threshold method based on DW is devised to enhance the open-set performance. By categorizing drifting synthetic samples into known and unknown classes and retraining them together with the known samples, the closed-set classifier is optimized, and a (K + 1)-class open-set classifier is trained. The experimental results in this research demonstrate the superior FSOSR performance of the proposed method across three benchmark HSI datasets.

Spectral-Spatial-Sensorial Attention Network with Controllable Factors for Hyperspectral Image Classification

Hyperspectral image (HSI) classification aims to recognize categories of objects based on spectral–spatial features and has been used in a wide range of real-world application areas. Attention mechanisms are widely used in HSI classification for their ability to focus on important information in images automatically. However, due to the approximate spectral–spatial features in HSI, mainstream attention mechanisms are difficult to accurately distinguish the small difference, which limits the classification accuracy. To overcome this problem, a spectral–spatial-sensorial attention network (S3AN) with controllable factors is proposed to efficiently recognize different objects. Specifically, two controllable factors, dynamic exponential pooling (DE-Pooling) and adaptive convolution (Adapt-Conv), are designed to enlarge the difference in approximate features and enhance the attention weight interaction. Then, attention mechanisms with controllable factors are utilized to build the redundancy reduction module (RRM), feature learning module (FLM), and label prediction module (LPM) to process HSI spectral–spatial features. The RRM utilizes the spectral attention mechanism to select representative band combinations, and the FLM introduces the spatial attention mechanism to highlight important objects. Furthermore, the sensorial attention mechanism extracts location and category information in a pseudo label to guide the LPM for label prediction and avoid details from being ignored. Experimental results on three public HSI datasets show that the proposed method is able to accurately recognize different objects with an overall accuracy (OA) of 98.69%, 98.89%, and 97.56%, respectively.

A Novel Transformer Network with a CNN-Enhanced Cross-Attention Mechanism for Hyperspectral Image Classification

Recently, with the remarkable advancements of deep learning in the field of image processing, convolutional neural networks (CNNs) have garnered widespread attention from researchers in the domain of hyperspectral image (HSI) classification. Moreover, due to the high performance demonstrated by the transformer architecture in classification tasks, there has been a proliferation of neural networks combining CNNs and transformers for HSI classification. However, the majority of the current methods focus on extracting spatial–spectral features from the HSI data of a single size for a pixel, overlooking the rich multi-scale feature information inherent to the data. To address this problem, we designed a novel transformer network with a CNN-enhanced cross-attention (TNCCA) mechanism for HSI classification. It is a dual-branch network that utilizes different scales of HSI input data to extract shallow spatial–spectral features using a multi-scale 3D and 2D hybrid convolutional neural network. After converting the feature maps into tokens, a series of 2D convolutions and dilated convolutions are employed to generate two sets of Q (queries), K (keys), and V (values) at different scales in a cross-attention module. This transformer with CNN-enhanced cross-attention explores multi-scale CNN-enhanced features and fuses them from both branches. Experimental evaluations conducted on three widely used hyperspectral image (HSI) datasets, under the constraint of limited sample size, demonstrate excellent classification performance of the proposed network.

Coupling randomisation and sparse modelling for the exploratory analysis of large hyperspectral datasets

Sparse-based models are a powerful tools for data compression, variable reduction, and model complexity reduction. Nevertheless, their major issue is the high computational time needed in large matrices. This manuscript proposes, for the first time, to couple randomised decomposition as a first step before sparsity calculations, followed by a projection of the full data onto a reduced-sparse set of loadings that will drastically reduce the time needed for calculations and built models that are equally reliable as their sparse-based homologous. While this new approach might be valid for several scenarios (exploration, regression and classification), we will focus on exploration methods (like Principal Component Analysis – PCA) applied to large datasets of hyperspectral images. Two datasets of different complexity have been tested, and the benefits of the coupled randomisation and sparse PCA (rsPCA) are extensively studied.

Graph-infused hybrid vision transformer: Advancing GeoAI for enhanced land cover classification

Hyperspectral Image Classification (HSIC) is a challenging task due to the high-dimensional nature of Hyperspectral Imaging (HSI) data and the complex relationships between spectral and spatial information. This paper proposes a Graph-Infused Hybrid spatial–spectral Transformer (GFormer) for HSIC. The GFormer combines the power of graph and spatial–spectral transformer to capture both spectral relationships and spatial context. We represent the HSI data as a graph, where nodes represent pixels and edges capture spectral similarities. By incorporating an attention mechanism, the GFormer learns spatial–spectral fusion representations, allowing it to effectively discriminate between different classes. The model can capture long-range dependencies among spectral bands, enabling it to understand complex interactions in the HSI data. Moreover, the GFormer adapts to different spectral resolutions by dynamically adjusting attention weights for each spectral band. Experimental results on benchmark HSI datasets demonstrate that the GFormer outperforms state-of-the-art (SOTA) methods, achieving superior classification accuracy.